Image processing method, device, equipment and computer readable storage medium

By calculating the overlap of image labeling candidate boxes and performing cross labeling merges, the problem of insufficient accuracy of labeling data caused by a single labeling personnel is solved, and the labeling accuracy and model recognition accuracy of endoscopic image data are improved.

CN111783863BActive Publication Date: 2024-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010581213.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-23
Publication Date
2024-11-12
Estimated Expiration
2040-07-31

AI Technical Summary

Technical Problem

In the prior art, a single labeler annotating image results in insufficient accuracy and stability of the labeling data, especially for complex endoscopic image data, the accuracy of model recognition cannot be guaranteed.

Method used

By obtaining at least two candidate images, the degree of overlap between candidate boxes is calculated, and the merge type is determined based on the degree of overlap, cross-labeling merge is performed, and the audit candidate results are generated to improve the accuracy of the label box, thereby improving the accuracy of model recognition.

Benefits of technology

The accuracy of the labeling frame of the target object in the labeling image is improved, the problem of insufficient individual capabilities of a single labeling person is compensated, the algorithm model is better learned of target features, and the model recognition accuracy is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111783863B_ABST
    Figure CN111783863B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an image processing method, apparatus, device and computer-readable storage medium, wherein the method comprises: obtaining at least two candidate images; at least two candidate images respectively carry candidate frames for marking target objects in the original image; determining the overlap between each two candidate frames in at least two candidate frames; determining the merge type corresponding to the at least two candidate frames according to the overlap, and determining the audit candidate results corresponding to the at least two candidate frames according to the merge type; sending the audit candidate results to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results; the annotated frame in the annotated image includes the target object. By adopting the present application, the accuracy of the annotated frame in the annotated image can be improved, and the accuracy of model recognition can be guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an image processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] In recent years, with the breakthrough of a new generation of artificial intelligence technology represented by deep learning, computer vision (CV) has made breakthrough progress. The algorithm model built with deep learning can not only classify the target objects in the image, but also determine the location information of the target objects. An excellent algorithm model relies on a large amount of representative high-quality annotated data to fully learn the target features. If the data annotation quality is poor, it is impossible to generate an algorithm model with strong stability and high accuracy.

[0003] However, in the existing technology, a single labeler labels a single image, resulting in the problem of low accuracy of the labeled data. In addition, since various types of endoscopic images have obvious individual differences and complexity, the labeling difficulty is greater than that of general image data. Therefore, the existing labeling method cannot guarantee the accuracy of model recognition. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device, and computer-readable storage medium, which can improve the accuracy of annotation boxes in annotated images, thereby ensuring the accuracy of model recognition.

[0005] An embodiment of the present application provides an image processing method, including:

[0006] Acquire at least two candidate images; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image;

[0007] Determine the overlap between every two candidate frames in at least two candidate frames;

[0008] Determine a merging type corresponding to at least two candidate boxes according to the degree of overlap, and determine candidate review results corresponding to at least two candidate boxes according to the merging type;

[0009] The audit candidate result is sent to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result; the annotated box in the annotated image includes the target object.

[0010] An embodiment of the present application provides an image processing method, including:

[0011] Acquire at least two candidate images and a labeling merging rule; the at least two candidate images respectively carry candidate frames for labeling target objects in the original image; the labeling merging rule refers to a rule for determining a merging type based on a degree of overlap between each two candidate frames in the at least two candidate frames;

[0012] By marking the merging type indicated by the merging rule, at least two candidate boxes are merged to obtain a review candidate result.

[0013] An embodiment of the present application provides an image processing device, including:

[0014] A first acquisition module is used to acquire at least two candidate images; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image;

[0015] A first determination module is used to determine the overlap between every two candidate frames in at least two candidate frames;

[0016] A second determination module is used to determine the merging type corresponding to the at least two candidate boxes according to the overlap degree, and determine the candidate review results corresponding to the at least two candidate boxes according to the merging type;

[0017] The result sending module is used to send the audit candidate results to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results; the annotated box in the annotated image includes the target object.

[0018] Wherein, the second determination module includes:

[0019] an effective merging unit, configured to select at least two effective candidate frames from the at least two candidate frames if the merging type is an effective merging type, merge the at least two effective candidate frames to obtain a merged candidate frame, and generate a review candidate result according to the merged candidate frame;

[0020] The first determination unit is used to determine at least two candidate images as review candidate results respectively if the merging type is an invalid merging type.

[0021] Wherein, the second determination module includes:

[0022] A second determination unit is used to determine the overlap degree greater than the overlap degree threshold as a valid overlap degree, and determine the number of valid overlap degrees;

[0023] A third determining unit, configured to determine that the merging type corresponding to at least two candidate frames is a valid merging type if the number of valid overlaps is equal to or greater than a number threshold;

[0024] The third determination unit is further configured to determine that the merging type corresponding to at least two candidate frames is an invalid merging type if the number of valid overlaps is less than a number threshold.

[0025] Among them, the effective merging units include:

[0026] A first determining subunit, configured to determine candidate frames corresponding to the effective overlap degree as at least two effective candidate frames;

[0027] The position acquisition subunit is used to respectively acquire position information of at least two valid candidate frames in the corresponding candidate images;

[0028] A merge position subunit, used to merge at least two pieces of position information to obtain merged position information;

[0029] The candidate generation subunit is used to generate a merging candidate box indicated by the merging position information.

[0030] The merge position subunit is specifically used to obtain the occurrence frequency of each valid candidate frame in the effective overlap degree, and determine the effective weight of each valid candidate frame according to the occurrence frequency;

[0031] The merged position subunit is further specifically used to perform weighted averaging on at least two pieces of position information according to effective weights to obtain merged position information.

[0032] The sending result module is specifically used for sending the original image carrying the merging candidate frame to the first audit terminal corresponding to the valid merging type if the merging type is a valid merging type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merging candidate frame;

[0033] The above-mentioned sending result module is also specifically used to send at least two candidate images to the second review terminal corresponding to the invalid merge type if the merge type is an invalid merge type, so that the second review terminal determines the annotated image corresponding to the original image based on the at least two candidate images.

[0034] Wherein, the first determination module includes:

[0035] Get the information unit to get the candidate box K i In the candidate image P i Location information L i , get the candidate box K j In the candidate image P j Location information L j ;

[0036] The fourth determining unit is configured to determine the position of the i And location information L j Determine the candidate box K iAnd the candidate box K j The overlap position information L ij ;

[0037] The fifth determining unit is configured to determine the position of the i , location information j And the coincidence position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0038] Wherein, the fifth determining unit includes:

[0039] The first acquisition subunit is used to acquire the location information L i For the candidate box K i Height H i and width W i , according to the height H i and width W i Determine the candidate box K i Area A i ;

[0040] The second acquisition subunit is used to acquire the location information L j For the candidate box K j Height H j and width W j , according to the height H j and width W j Determine the candidate box K j Area A j ;

[0041] The third acquisition subunit is used to obtain the overlap position information L ij In the example, for the candidate box K i And the candidate box K j The overlap height H ij And the overlap width W ij , according to the overlap height H ij And the overlap width W ij , determine the candidate box K i And the candidate box K j The overlap area A ij ;

[0042] Determine the overlapping subunits for the area A i 、Area A j And the overlap area A ij , determine the candidate box K i And the candidate box K j The overlap betweenij .

[0043] The image processing device further includes:

[0044] The second acquisition module is used to acquire a training sample set; the training sample set includes annotated images and original images;

[0045] The second acquisition module is further used to determine the position information of the annotation box in the annotated image as the position label, and determine the object attribute category of the target object in the annotated image as the category label;

[0046] The second acquisition module is also used to input the original image into the initial image recognition model;

[0047] A third acquisition module is used to acquire image features of the original image according to the initial image recognition model, and predict the predicted object attribute category and predicted position information of the target object according to the image features;

[0048] The third acquisition module is also used to adjust the initial image recognition model according to the predicted object attribute category, predicted location information, category label and location label to obtain the image recognition model.

[0049] Wherein, the third acquisition module includes:

[0050] A sixth determining unit, used to determine a category loss value according to the predicted object attribute category and the category label;

[0051] A sixth determining unit is further used to determine a position loss value according to the predicted position information and the position tag;

[0052] The sixth determining unit is further used to determine the total loss value according to the category loss value and the position loss value;

[0053] The seventh determination unit is used to adjust the weights of the parameters in the initial image recognition model according to the total loss value, and when the total loss value meets the convergence condition, the adjusted image recognition model is determined as the image recognition model.

[0054] The image processing device further includes:

[0055] A fourth acquisition module is used to acquire at least two sample annotated images; the at least two sample annotated images include an annotated image;

[0056] The fourth acquisition module is further used to add, among the at least two sample annotated images, the sample annotated images associated with the valid merge type to the first image set, and add the sample annotated images associated with the invalid merge type to the second image set; the first image set has a first sampling probability, the second image set has a second sampling probability, and the first sampling probability is greater than the second sampling probability;

[0057] The fourth acquisition module is further used to collect sample annotated images between the first image set and the second image set according to the first sampling probability and the second sampling probability, and add the collected sample annotated images to the training sample set.

[0058] Wherein, the image processing device further includes:

[0059] A fifth acquisition module is used to acquire a medical detection image; the medical detection image includes an object to be identified;

[0060] The fifth acquisition module is further used to input the medical detection image into the image recognition model, and obtain the medical image features of the medical detection image in the image recognition model;

[0061] The fifth acquisition module is also used to output the object detection category and the object location information according to the medical image features in the image recognition model;

[0062] The fifth acquisition module is further used to generate a medical annotation frame according to the object position information; the medical annotation frame is used to mark the object to be identified in the medical detection image;

[0063] The fifth acquisition module is also used to display the medical annotation box and the object detection category in the medical detection image.

[0064] An embodiment of the present application provides an image processing device, including:

[0065] An acquisition rule module is used to acquire at least two candidate images and an annotation merging rule; the at least two candidate images respectively carry candidate frames for annotating target objects in the original image; the annotation merging rule refers to a rule for determining a merging type based on the overlap between each two candidate frames in the at least two candidate frames;

[0066] The candidate merging module is used to merge at least two candidate boxes by marking the merging type indicated by the merging rule to obtain the audit candidate result.

[0067] Among them, the merging candidate module is specifically used to merge the three candidate boxes by marking the valid merging type indicated by the merging rule, obtain the merged candidate box, and generate the review candidate result according to the merged candidate box.

[0068] Among them, the merge candidate module is specifically used to merge two candidate boxes whose overlap is greater than the overlap threshold by marking the valid merge type indicated by the merge rule, obtain a merged candidate box, and generate an audit candidate result based on the merged candidate box.

[0069] Among them, the merging candidate module is specifically used to determine the three candidate images as the review candidate results by marking the invalid merging types indicated by the merging rules.

[0070] Among them, the merge candidate module is specifically used to merge two candidate boxes by marking the valid merge type indicated by the merge rule, obtain the merged candidate box, and generate the review candidate result according to the merged candidate box.

[0071] Among them, the merging candidate module is specifically used to determine two candidate images as review candidate results by marking the invalid merging types indicated by the merging rules.

[0072] On one hand, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0073] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide a data communication function, the above-mentioned memory is used to store a computer program, and the above-mentioned processor is used to call the above-mentioned computer program to execute the method in the embodiment of the present application.

[0074] On one hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0075] On the one hand, the embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the embodiments of the present application.

[0076] In the embodiment of the present application, by acquiring at least two candidate images, at least two candidate frames that respectively carry the target object for annotating the original image can be acquired, and the overlap between each two candidate frames in the at least two candidate frames can be determined. The merge type corresponding to the at least two candidate frames can be determined based on the overlap, and then the audit candidate results corresponding to the at least two candidate frames can be determined based on the merge type, and the audit candidate results can be sent to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image based on the audit candidate results, and the annotated frame in the annotated image includes the target object. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotator by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotated frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, and can improve the recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1a It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0079] Figure 1b is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0080] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0081] Figure 3 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0082] Figure 4 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0083] Figure 5 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0084] Figure 6 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0085] Figure 7a is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0086] Figure 7b is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0087] Figure 8 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0088] Fig. 9 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0089] Fig.10 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0090] Fig.11 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0091] Fig.12is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0092] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0094] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0095] The solution provided in the embodiments of the present application involves artificial intelligence computer vision technology and deep learning technology, which is specifically described by the following embodiments:

[0096] See also Figure 1a , Figure 1a Schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1aAs shown, the system may include an image database 10a, a target terminal 20a, a first audit terminal 20b, and a second audit terminal 20c. The target terminal 20a, the first audit terminal 20b, and the second audit terminal 20c may all obtain at least two candidate images from the image database 10a. The target terminal 20a is connected to the first audit terminal 20b, the target terminal 20a is connected to the second audit terminal 20c, and the first audit terminal 20b and the second audit terminal 20c may also be connected to each other. The image database 10a may be composed of image data included in the business terminal 10b, image data included in the business terminal 10c, ..., image data included in the business terminal 10d.

[0097] Please also see Figure 1b , Figure 1b Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 1b As shown, the annotator 30B opens and views the original image S through the annotation application software A installed in the service terminal 10b, and annotates the target object in the original image S, such as Figure 1b The candidate box shown in , can obtain the candidate image B, the above original image S carries the basic information such as the image name, the image identification code, and the image identification code is unique. In this embodiment, the colorectal annotation is used as an example to illustrate the image name, and the patient number 123854 is used as an example to illustrate the image identification code. In actual application, it can be changed according to the application scenario; the candidate image B carries the basic information such as the image name, the image identification code, the independent annotation result, the annotation person information and the independent annotation result number. The independent annotation result can refer to Figure 1b The candidate box shown in the figure may also be the position information of the above-mentioned candidate box, which is not limited here. This embodiment takes khy752 as an example to illustrate the annotation personnel information. This embodiment takes the result number 5588943 as an example to name the independent annotation result number, which is also required to be unique. When the annotation personnel 30B triggers the "Submit" button, the business terminal 10b transmits the original image S and the candidate image B to the image database 10a as the image data in the image database 10a.

[0098] Optionally, other annotators can also annotate the original image S through the annotation application software A installed in their respective business terminals (such as business terminal 10c and business terminal 10d) to obtain their respective candidate images. Similarly, each candidate image can carry its own image name, image identification code, independent annotation result, annotator information, independent annotation result number and other basic information. The original image S and these candidate images are stored in the image database 10a through data transmission. According to the image identification code of the original image S, the image database 10a can store these candidate images in association with the candidate image B. The business terminals (including business terminal 10b, business terminal 10c, ..., business terminal 10d) and the image database 10a can directly or indirectly transmit data through wired or wireless communication, and this application does not limit this.

[0099] Considering that the label quality of a single candidate image cannot be guaranteed, the present application embodiment proposes an image processing process, which is divided into four steps: independent labeling by at least two labelers, cross-labeling merging, arbitration labeling, and review labeling. Figure 1a , each process involved in the image processing is introduced in detail.

[0100] First step: independent labeling, please refer to Figure 1b , the annotator 30B checks the original image A through the service terminal 10b to confirm whether the original image A contains the target object. If so, the target object is annotated in the form of a box selection to form a candidate box and obtain a candidate image B; if the annotator 30B confirms that the original image A does not contain the target object, the annotator can give up the annotation of the original image A. Other annotators also annotate the target object in the original image A through the above process. At least two annotators independently annotate the same original image, and at least two candidate images for the original image can be obtained, and then at least two independent candidate boxes can be obtained.

[0101] Second process: cross-annotation merging. In order to ensure the quality of image annotation and reduce the individual differences between annotators, the target terminal 20a obtains the overlap degree of at least two candidate frames generated by the first process. Take candidate image B and candidate image C (candidate image B and candidate image C are both associated with the same original image) as an example. Candidate image B includes candidate frame K b , the candidate image C includes the candidate box K c , the target terminal 10a can calculate the candidate box K b With the candidate box K c The overlap between them can be calculated by the intersection over union (IOU) b With the candidate box K cThe overlap between them is calculated as shown in formula (1):

[0102]

[0103] That is, calculate the candidate box K b With the candidate box K c The ratio of the intersection to the union, IOU value ranges from 0 to 1. The larger the IOU value, the higher the overlap of the two candidate boxes. When IOU is 0, the two candidate boxes do not overlap and have no intersection. When IOU is 1, the two candidate boxes completely overlap. The ideal situation is complete overlap, that is, the ratio is 1.

[0104] When a cross-annotation merge is obtained based on at least two candidate images, the possible situations and subsequent processes are as follows:

[0105] 1) Three independent annotation results;

[0106] The following description is made by taking the labeling personnel 30B, the labeling personnel 30C, and the labeling personnel 30D as examples. The labeling personnel 30B labels the original image S through the service terminal 10b to obtain the candidate image B. The candidate image B includes the candidate box K b ; The annotator 30C annotates the original image S through the service terminal 10c to obtain a candidate image C, which includes a candidate box K c ; The annotator 30D annotates the original image S through the service terminal 10d to obtain a candidate image D, which includes a candidate box K d .

[0107] After the target terminal 20a obtains the candidate images B, C, and D from the image database 10a, it obtains the candidate frame K of the candidate image B. b , candidate box K of candidate image C c And the candidate box K of the candidate image D d , calculate the candidate box K b Compared with the candidate box K c The overlap between bc , calculate the candidate box K c Compared with the candidate box K d The overlap between cd , calculate the candidate box K b Compared with the candidate box K d The overlap between bd , after the calculation is completed, there are four situations:

[0108] All three overlaps are greater than the overlap threshold, i.e. IOU bc , IOU bd and IOUcd The target terminal 20a selects three candidate frames (candidate frame K b , candidate box K c And the candidate box K d ) to obtain the merged candidate box K bcd , merge the candidate box K bcd Add to the original image as a candidate audit result, and send the candidate audit result to the first audit terminal 20b;

[0109] If the two overlaps are greater than the overlap threshold, bd and IOU cd is greater than the overlap threshold, and IOU bc is less than or equal to the overlap threshold, the target terminal 20a selects three candidate frames (candidate frame K b , candidate box K c And the candidate box K d ) to obtain the merged candidate box K bcd , merge the candidate box K bcd Add to the original image, obtain the audit candidate result, and send it to the first audit terminal 20b;

[0110] Only one overlap is greater than the overlap threshold, if IOU bd is greater than the overlap threshold, and IOU bc and IOU cd is less than or equal to the overlap threshold, the target terminal 20a selects the candidate box K b And the candidate box K d Merge and get the merged candidate box K bd , merge the candidate box K bd Add to the original image to obtain the audit candidate result, and the target terminal 20a sends the audit candidate result to the first audit terminal 20b;

[0111] If the three coincidences are all less than or equal to the coincidence threshold, the target terminal 20a determines the candidate images B, C and D as the candidate audit results, and sends the candidate audit results to the second audit terminal 20c.

[0112] 2) Two independent annotation results;

[0113] The following is an example of a labeler 30B and a labeler 30C. The labeler 30B labels the original image S through the service terminal 10b to obtain a candidate image B. The candidate image B includes a candidate box K b ; The annotator 30C annotates the original image S through the service terminal 10c to obtain a candidate image C, which includes a candidate box K c .

[0114] After the target terminal 20a obtains the candidate images B and C from the image database 10a, it selects the candidate frame K of the candidate image B. b , candidate box K of candidate image C c , calculate the candidate box K b Compared with the candidate box K c The overlap between bc , at this time, there are the following two situations:

[0115] ·IOU bc If the overlap degree is greater than the threshold, the target terminal 20a selects the candidate box K b And the candidate box K c Merge and get the merged candidate box K bc , merge the candidate box K bc Added to the original image, and sent to the first audit terminal 20b as an audit candidate result;

[0116] ·IOU bc If the overlap degree is less than or equal to the overlap degree threshold, the target terminal 20a sends the candidate image B and the candidate image C as the audit candidate results to the second audit terminal 20c.

[0117] 3) There is only one annotation result

[0118] Abandon and do not proceed with the subsequent process.

[0119] The third process: review and annotation. This process occurs in the first review terminal 20b. If the review candidate result obtained in the second process is the original image carrying the merge candidate box, the first review terminal 20b can receive the original image carrying the merge candidate box sent by the target terminal 20a. Then the reviewer corresponding to the first review terminal 20b confirms the review candidate result. If the merge candidate box is approved, the original image carrying the merge candidate box is determined as the annotated image. If the merge candidate box is not approved, the original image carrying the merge candidate box is forwarded to the second review terminal 20c.

[0120] The fourth process: arbitration annotation, which occurs in the second audit terminal 20c. If the audit candidate results obtained in the second process are the three candidate images mentioned above (candidate image B, candidate image C, and candidate image D), the second audit terminal 20c can receive the audit candidate results sent by the target terminal 20a, and then the auditor corresponding to the second audit terminal 20c annotates the original image, and determines the original image with the annotation box as the annotated image, and then sends the annotated image to the first audit terminal 20c for review and storage. Secondly, the second audit terminal 20c can view the original image with the merged candidate box forwarded by the first audit terminal 20b, and annotate the original image. Similarly, the original image with the annotation box is determined as the annotated image. The subsequent steps are consistent with the above and will not be repeated here.

[0121] As described above, the embodiment of the present application can make up for the problem of excessive fluctuation in annotation quality caused by individual ability differences of annotation personnel through four processes: independent annotation by at least two annotation personnel, cross-annotation merging, arbitration annotation, and review annotation, thereby improving the accuracy and reliability of annotation data.

[0122] It should be understood that in the embodiment of the present application, the labelers who annotate the original image may include multiple people, not just the three or two mentioned above. For example, the original image S is assigned to labeler 1a, labeler 2a, labeler 3a, labeler 4a, and labeler 5a. The possible labeling results are: labelers 1a-3a believe that there is a target object in the original image S, so they select the target object in the original image S, while labelers 4a and 5a believe that there is no target object in the original image S, so they give up labeling the original image S.

[0123] The image database 10a can be regarded as an electronic file cabinet, a place for storing electronic files (image files in this application), where terminal users can add, query, update, delete, etc. images in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application.

[0124] It is understandable that the method provided in the embodiment of the present application can be performed by a computer device, and the computer device includes but is not limited to a terminal or a server. The target terminal 20a in the embodiment of the present application can be a computer device. The above-mentioned server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0125] in, Figure 1a The business terminal 10b, business terminal 10c, business terminal 10d, target terminal 20a, first audit terminal 20b and second audit terminal 20c may include mobile phones, tablet computers, laptop computers, PDAs, smart speakers, mobile Internet devices (MID), POS (Point Of Sales), wearable devices (such as smart watches, smart bracelets, etc.), etc.

[0126] For further information, see Figure 2 , Figure 2 is a flow chart of an image processing method provided in an embodiment of the present application. Figure 2 As shown, the image processing process includes the following steps:

[0127] Step S101, obtaining at least two candidate images; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image.

[0128] For details, please refer to Figure 3 , Figure 3 Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 3 As shown, in the embodiment of the present application, the target terminal 20a is used as the execution subject, and the number of at least two candidate images is set to 3, that is, Figure 3 In the candidate images 101a, 101b and 101c, if the number of at least two candidate images is 2 or other numbers, this embodiment can be referred to, such as Figure 3 As shown, the original image A includes objects such as houses, pedestrians, escalators, and buildings. In this embodiment, the target object in the original image A is set to be a pedestrian.

[0129] The target terminal 20a obtains candidate images 101a, 101b, and 101c. The above three candidate images are all generated based on the original image A. For example, candidate image 101a is an image generated by annotating personnel 101A annotating pedestrians (target objects) in the original image A, candidate image 101b is an image generated by annotating personnel 101B annotating pedestrians in the original image A, and candidate image 101c is an image generated by annotating personnel 101C annotating pedestrians in the original image A. Figure 3 As shown, candidate image 101a carries candidate frame 102a, candidate image 101b carries candidate frame 102b, and candidate image 101c carries candidate frame 102c.

[0130] The original image can carry basic information such as image name, image identification code, acquisition device model, acquisition date, image size, acquisition frame rate, and acquisition location. The image identification code is unique, so the target terminal 20a can obtain the original image and the candidate image corresponding to the original image based on the image identification code.

[0131] In addition to the basic information of the original image itself, at least two candidate images also carry basic information about the candidate box and the annotation personnel. For example, the annotation personnel 101A annotates the original image A for the target object through the business terminal to generate the candidate image 101a. Then the candidate image 101a can carry the independent annotation result, the annotation personnel information, the annotation date and the independent annotation result number. The above-mentioned independent annotation result can refer to the visualized candidate box 102a, or it can refer to the position information of the candidate box 102a in the candidate image 101a. The above-mentioned independent annotation result number is unique, and the target terminal 20a can obtain the candidate image 101a according to the independent annotation result number.

[0132] It is understandable that the at least two candidate images mentioned above can be images stored locally in the target terminal 20a, or images sent to the target terminal 20a by other terminals, or images obtained by the target terminal 20a from an image database. The source of the at least two candidate images is not limited here.

[0133] Step S102: determining the degree of overlap between every two candidate frames in at least two candidate frames.

[0134] Specifically, the at least two candidate images include candidate image P i And the candidate image P j , candidate image P i Carry candidate box W i , candidate image P j Carry candidate box W j, where i and j are positive integers, and i and j are less than or equal to the total number of candidate images in at least two candidate images; obtain the candidate frame K i In the candidate image P i Location information L i , get the candidate box K j In the candidate image P j Location information L j ; According to the location information L i And location information L j Determine the candidate box K i And the candidate box K j The overlap position information L ij ; Get location information L i For the candidate box K i Height H i and width W i , according to the height H i and width W i Determine the candidate box K i Area A i ; Get location information L j For the candidate box K j Height H j and width W j , according to the height H j and width W j Determine the candidate box K j Area A j ; Get the overlap position information L ij In the example, for the candidate box K i And the candidate box K j The overlap height H ij And the overlap width W ij , according to the overlap height H ij And the overlap width W ij , determine the candidate box K i And the candidate box K j The overlap area A ij ; According to the area A i 、Area A j And the overlap area A ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0135] In order to make up for the defect of unstable annotation quality of a single labeled image, the embodiment of the present application enables multiple annotators to independently annotate the same original image to obtain multiple candidate images. Figure 3 , as can be seen from step S101, Figure 3 The three candidate images (i.e., candidate image 101a, candidate image 101b, and candidate image 101c) in the figure can be regarded as the label images of the original image A. Except for the difference in the candidate frames they carry, there is no other difference between the three images (because the three candidate images are all generated based on the original image A). Therefore, the upper left corner of the above four images (i.e., the original image A, candidate image 101a, candidate image 101b, and candidate image 101c) is the coordinate origin, the x-axis is to the right of the origin, and the y-axis is to the bottom of the origin. The coordinates generated are consistent. Therefore, the four position information corresponding to the target objects in the four images are consistent.

[0136] like Figure 3 As shown, the target terminal 20a can superimpose the candidate frame 102b in the candidate image 101b onto the candidate image 101a. Obviously, there are overlapping areas and different areas between the candidate frames 102a and 102b. With the upper left corner of the candidate image 101a as the coordinate origin (0,0), the x-axis is to the right of the origin, and the y-axis is to the bottom of the origin. The target terminal 20a obtains the coordinates of the center point of the candidate frame 102a in the candidate image 101a and the coordinates of the four frame corners (i.e., the upper left corner, the lower left corner, the upper right corner, and the lower right corner), and can obtain the position information 103a of the candidate frame 102a in the candidate image 101a, i.e., (x a ,y a ,w a ,h a ), where x a refers to the horizontal coordinate of the center point of the candidate box 102a in the candidate image 101a, y a refers to the vertical coordinate of the center point of the candidate frame 102a in the candidate image 101a, w a refers to the width of the candidate box 102a, h a refers to the height of the candidate frame 102a; the target terminal 20a obtains the center point coordinates and the four frame corner coordinates of the candidate frame 102b in the candidate image 101a, and can obtain the position information 103b of the candidate frame 102b in the candidate image 101a, that is, (x b ,y b ,w b ,h b ), where x b refers to the horizontal coordinate of the center point of the candidate box 102b in the candidate image 101a, y b refers to the vertical coordinate of the center point of the candidate box 102b in the candidate image 101a, b refers to the width of the candidate box 102b, h b It refers to the height of the candidate box 102b.

[0137] The target terminal 20a can determine the overlapping position information 103ab between the candidate frame 102a and the candidate frame 102b according to the position information 103a and the position information 103b, that is, (x ab ,y ab ,w ab ,h ab ), according to the location information 103a, the target terminal 20a can determine the area 104a of the candidate box 102a, that is, (w a *h a ), according to the location information 103b, the target terminal 20a can determine the area 104b of the candidate box 102b, that is, (w b *h b ), according to the overlap position information 103ab, the target terminal 20a can determine the overlap area 104ab of the overlap candidate frame 102ab, that is, (w ab *h ab ), the overlapping candidate box 102ab refers to the overlapping area between the candidate box 102a and the candidate box 102b. Figure 4 , Figure 4 Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 4 As shown in FIG. 2 , the target terminal 20a can determine the overlap degree C between the candidate frame 102a and the candidate frame 102b according to formula (2): ab .

[0138]

[0139] Please see again Figure 3 , the target terminal 20a can superimpose the candidate frame 102c in the candidate image 101c onto the candidate image 101a. Obviously, there are overlapping areas and different areas between the candidate frames 102a and 102c. With the upper left corner of the candidate image 101a as the coordinate origin (0,0), the x-axis is to the right of the origin, and the y-axis is to the bottom of the origin. The target terminal 20a obtains the center point coordinates and the four frame corner coordinates of the candidate frame 102c in the candidate image 101a, and can obtain the position information 103c of the candidate frame 102c in the candidate image 101a, that is, (x c ,y c ,w c ,h c ), where x c It refers to the horizontal coordinate of the center point of the candidate box 102c in the candidate image 101a, y c refers to the vertical coordinate of the center point of the candidate box 102c in the candidate image 101a, c refers to the width of the candidate box 102c, h c It refers to the height of the candidate box 102c.

[0140] The target terminal 20a can determine the overlapping position information 103ac between the candidate frame 102a and the candidate frame 102c according to the position information 103a and the position information 103c, that is, (x ac ,y ac ,w ac ,h ac ), according to the location information 103a, the target terminal 20a can determine the area 104a of the candidate box 102a, that is, (w a *h a ), according to the location information 103c, the target terminal 20a can determine the area 104c of the candidate box 102c, that is, (w c *h c ), according to the overlap position information 103ac, the target terminal 20a can determine the overlap area 104ac of the overlap candidate frame 102ac, that is, (w ac *h ac ), the overlapping candidate box 102ac refers to the overlapping area between the candidate box 102a and the candidate box 102c. Figure 4 The target terminal 20a can determine the overlap degree C between the candidate box 102a and the candidate box 102c according to formula (3). ac .

[0141]

[0142] Please see again Figure 3 , the target terminal 20a can superimpose the candidate frame 102c in the candidate image 101c onto the candidate image 101b. Obviously, there are overlapping areas and different areas between the candidate frames 102b and 102c. With the upper left corner of the candidate image 101b as the coordinate origin (0,0), the x-axis is to the right of the origin, and the y-axis is to the bottom of the origin. The target terminal 20a obtains the center point coordinates and the four frame corner coordinates of the candidate frame 102b in the candidate image 101b, and can obtain the position information 103b of the candidate frame 102b in the candidate image 101b, that is, (x b ,y b ,w b ,h b ), where x b refers to the horizontal coordinate of the center point of the candidate box 102b in the candidate image 101b, y b refers to the vertical coordinate of the center point of the candidate box 102b in the candidate image 101b, b refers to the width of the candidate box 102b, h bThe target terminal 20a obtains the center point coordinates and the four corner coordinates of the candidate frame 102c in the candidate image 101b, and can obtain the position information 103c of the candidate frame 102c in the candidate image 101b, that is, (x c ,y c ,w c ,h c ), where x c It refers to the horizontal coordinate of the center point of the candidate box 102c in the candidate image 101b, y c refers to the vertical coordinate of the center point of the candidate box 102c in the candidate image 101b, c refers to the width of the candidate box 102c, h c It refers to the height of the candidate box 102c.

[0143] The target terminal 20a can determine the overlapping position information 103bc between the candidate frame 102b and the candidate frame 102c according to the position information 103b and the position information 103c, that is, (x bc ,y bc ,w bc ,h bc ), according to the location information 103b, the target terminal 20a can determine the area 104b of the candidate box 102b, that is, (w b *h b ), according to the location information 103c, the target terminal 20a can determine the area 104c of the candidate box 102c, that is, (w c *h c ), according to the overlap position information 103bc, the target terminal 20a can determine the overlap area 104bc of the overlap candidate frame 102bc, that is, (w bc *h bc ), the overlapping candidate box 102bc refers to the overlapping area between the candidate box 102b and the candidate box 102c. Figure 4 The target terminal 20a can determine the overlap degree C between the candidate box 102b and the candidate box 102c according to formula (4). bc .

[0144]

[0145] Optionally, the target terminal 20a may superimpose the candidate image 101a and the candidate image 101b, or may determine the overlap area between the two candidate frames, and further determine the overlap degree between the two candidate frames.

[0146] The candidate box W mentioned above i And the candidate box W j It can be any one of the three situations described above.

[0147] Step S103, determining the merging type corresponding to at least two candidate boxes according to the degree of overlap, and determining the review candidate results corresponding to at least two candidate boxes according to the merging type.

[0148] Specifically, the overlap greater than the overlap threshold is determined as a valid overlap, and the number of valid overlaps is determined; if the number of valid overlaps is equal to or greater than the number threshold, the merge type corresponding to at least two candidate frames is determined to be a valid merge type; if the number of valid overlaps is less than the number threshold, the merge type corresponding to at least two candidate frames is determined to be an invalid merge type. If the merge type is a valid merge type, the candidate frames corresponding to the valid overlap are determined to be at least two valid candidate frames, and the position information of the at least two valid candidate frames in the candidate images to which they belong is obtained respectively; the frequency of occurrence of each valid candidate frame in the valid overlap is obtained, and the effective weight of each valid candidate frame is determined according to the frequency of occurrence, and at least two position information are weighted averaged according to the effective weight to obtain merged position information, and a merged candidate frame indicated by the merged position information is generated; the merged candidate frame is added to the original image to obtain a candidate result for review; if the merge type is an invalid merge type, at least two candidate images are determined as candidate results for review respectively.

[0149] From step S102, it can be seen that the target terminal 20a can obtain three overlaps, namely, overlap C ab , overlap degree C ac , overlap degree C bc , please see Figure 4 , the overlap degree C ab Compare with the overlap threshold. If the overlap C ab is greater than the overlap threshold, the target terminal 20a determines the overlap C ab is the effective overlap, if the overlap C ab If it is less than or equal to the overlap threshold, the overlap C is determined. ab The target terminal 20a will be overlapped C ac Compare with the overlap threshold. If the overlap C ac If it is greater than the overlap threshold, the overlap C is determined. ac is the effective overlap, if the overlap C ac If it is less than or equal to the overlap threshold, the overlap C is determined. ac The target terminal 20a will be overlapped C bc Compare with the overlap threshold. If the overlap C bc If it is greater than the overlap threshold, the overlap C is determined. bc is the effective overlap, if the overlap C bc If it is less than or equal to the overlap threshold, the overlap C is determined. bcInvalid overlap.

[0150] like Figure 4 As shown, the overlap degree C ab , overlap degree C ac , overlap degree C bc They are all greater than the overlap threshold, so the target terminal 20a determines that the number of valid overlaps is 3, and compares the valid overlap number 3 with the quantity threshold. If the valid overlap number 3 is equal to or greater than the quantity threshold (that is, the quantity threshold is equal to or less than 3), the merge type of the three candidate boxes is determined to be a valid merge type; if the valid overlap number 3 is less than the quantity threshold (that is, the quantity threshold is greater than 3), the merge type of the three candidate boxes is determined to be an invalid merge type.

[0151] When the merge type of the three candidate boxes is a valid merge type, see Figure 5 , Figure 5 Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 5 As shown, the overlap degree C ab , overlap degree C ac , overlap degree C bc All of them are valid overlaps, so the target terminal 20a can determine that the candidate boxes 102a, 102b and 102c are all valid candidate boxes, and thus first obtain the position information corresponding to the candidate boxes 102a, 102b and 102c, that is, (x a ,y a ,w a ,h a )、(x b ,y b ,w b ,h b ) and (x c ,y c ,w c ,h c ), and then obtain the frequency of occurrence of each valid candidate frame in the effective overlap, and the target terminal 20a calculates the frequency of occurrence of each valid candidate frame in the effective overlap according to the overlap C ab , overlap degree C ac , overlap degree C bc, it can be seen that there are 6 valid candidate frames, namely 2 candidate frames 102a, 2 candidate frames 102b and 2 candidate frames 102c. Therefore, the appearance frequency of candidate frame 102a in the effective overlap is 1 / 3, the appearance frequency of candidate frame 102b in the effective overlap is 1 / 3, and the appearance frequency of candidate frame 102c in the effective overlap is 1 / 3. Then, the target terminal 20a can determine that the effective weight of candidate frame 102a is 1 / 3, the effective weight of candidate frame 102b is 1 / 3, and the effective weight of candidate frame 102c is 1 / 3. According to the above 3 effective weights and 3 position information, the target terminal 20a can refer to formula (5) to obtain the merged position information 105abc.

[0152] 105abc=1 / 3(x a ,y a ,w a ,h a )+1 / 3(x b ,y b ,w b ,h b )+1 / 3(x c ,y c ,w c ,h c ) (5)

[0153] The target terminal 20a generates a merge candidate frame 106a indicated by the merge position information, adds the merge candidate frame 106a to the original image, and generates a review candidate image 106, such as Figure 5 As shown, the above-mentioned audit candidate image 106 is used as the audit candidate result. In addition to the basic information of the original image itself, the audit candidate image 106 may also carry basic information such as the merged annotation result, the merged annotation result number, the merge type, and the steering type.

[0154] If the merge type is an invalid merge type, the target terminal 20a determines the candidate image 101a, the candidate image 101b, and the candidate image 101c as the audit candidate results.

[0155] Step S104, sending the audit candidate results to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results; the annotated box in the annotated image includes the target object.

[0156] Specifically, the review terminal includes a first review terminal and a second review terminal; if the merge type is a valid merge type, the original image carrying the merge candidate box is sent to the first review terminal corresponding to the valid merge type, so that the first review terminal determines the annotated image corresponding to the original image based on the merge candidate box; if the merge type is an invalid merge type, at least two candidate images are sent to the second review terminal corresponding to the invalid merge type, so that the second review terminal determines the annotated image corresponding to the original image based on the at least two candidate images; the annotated box in the annotated image includes the target object.

[0157] If the merge type is invalid, please refer to Figure 6 , Figure 6 Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 6 As shown, the target terminal 20a sends at least two candidate images to the second audit terminal 20c, and the second audit terminal 20c has the function of arbitration in the entire image processing process. Since the labeling personnel 101A, the labeling personnel 101B and the labeling personnel 101C all believe that there is a pedestrian (i.e., the target object) in the original image A, but the overlap between the candidate boxes 102a, the candidate boxes 102b and the candidate boxes 102c is less than the overlap threshold, the target terminal 20a cannot merge the candidate boxes 102a, the candidate boxes 102b and the candidate boxes 102c. At this time, the arbitration personnel 40C corresponding to the second audit terminal 20c labels the pedestrian in the original image A, obtains the labeling box 107a, generates the labeling image 107, and then the second audit terminal 20c sends the labeling image 107 to the first audit terminal 20b. After the first audit terminal 20b obtains the labeling image 107, it saves it. In addition to the basic information of the original image A, the annotated image 106 may also carry the arbitration annotation result, the arbitration annotation result number, the arbitration personnel information, the arbitration date, and the arbitration result number. The annotated image 107 may be used as a training sample to train the network model.

[0158] If the merge type is a valid merge type, please also refer to Figure 7a , Figure 7a Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 7aAs shown, the target terminal 20a sends the audit candidate result (i.e., the audit candidate image 106 described in step S103) to the first audit terminal 20b. The first audit terminal 20b has the audit function in the entire image processing process. After the first audit terminal 20b obtains the audit candidate image 106, the auditor 40B can audit the image through the first audit terminal 20b. If the auditor 40B approves the audit candidate image 106, the first audit terminal 20b can save it. If the audit candidate image 106 is rejected, the audit candidate image 106 can be forwarded to the second audit terminal 20c through the first audit terminal 20b. At this time, the audit candidate image 106 can carry the auditor information, audit date, and audit result number in addition to the basic information of the original image A itself. Please refer to the arbitration process of the second audit terminal 20c for the subsequent process, which will not be repeated here.

[0159] It should be noted that if the target terminal 20a obtains only one candidate image, the candidate image is discarded.

[0160] Optionally, obtain at least two original images, such as original image 1, original image 2,…, original image N, where N is an integer greater than 1, and let multiple annotators independently annotate the target object in each original image. Therefore, multiple candidate images 1a (each candidate image includes a candidate box) for original image 1, multiple candidate images 2a for original image 2,…, and multiple candidate images Na for candidate image N can be obtained. According to the above steps S101 to S104, the candidate boxes respectively included in the multiple candidate images 1a can obtain the annotated image 1b for the original image 1, the candidate boxes respectively included in the multiple candidate images 2a can obtain the annotated image 2b for the original image 2, ..., the candidate boxes respectively included in the multiple candidate images Na can obtain the annotated image Nb for the original image N, and the above at least two annotated images are determined as at least two sample annotated images, and the at least two sample annotated images include an annotated image; among the at least two sample annotated images, the sample annotated images associated with the valid merge type are added to the first image set, and the sample annotated images associated with the invalid merge type are added to the second image set, the first image set has a first sampling probability, and the second image set has a second sampling probability, and the first sampling probability is greater than the second sampling probability; according to the first sampling probability and the second sampling probability, sample annotated images are collected between the first image set and the second image set, and the collected sample annotated images are added to the training sample set.

[0161] A training sample set is obtained, which includes an annotated image and an original image; the position information of the annotation box in the annotated image is determined as the position label, and the object attribute category of the target object in the annotated image is determined as the category label; the original image is input into the initial image recognition model; the image features of the original image are obtained according to the initial image recognition model, and the predicted object attribute category and predicted position information of the target object are predicted according to the image features; the category loss value is determined according to the predicted object attribute category and the category label; the position loss value is determined according to the predicted position information and the position label; the total loss value is determined according to the category loss value and the position loss value; the weights of the parameters in the initial image recognition model are adjusted according to the total loss value, and when the total loss value meets the convergence condition, the adjusted image recognition model is determined as the image recognition model.

[0162] Acquire a medical test image; the medical test image includes an object to be identified; input the medical test image into an image recognition model, and obtain medical image features of the medical test image in the image recognition model; in the image recognition model, output object detection categories and object location information according to the medical image features; generate a medical annotation box according to the object location information; the medical annotation box is used to mark the object to be identified in the medical test image; in the medical test image, display the medical annotation box and the object detection category.

[0163] Please also see Figure 7b , Figure 7b Schematic diagram of an image processing scenario provided by an embodiment of the present application. Figure 7b As shown, the target terminal 70h obtains at least two sample annotated images 70a, each of which is generated by at least two candidate images. It can be seen from step S103 that each sample annotated image corresponds to two merging types. Please refer to the above for the specific determination process, which will not be repeated here. If the annotated image is associated with a valid merging type, the annotation box in the annotated image is generated by merging at least two valid candidate boxes. If the annotated image is associated with an invalid merging type, the annotation box is not generated by merging at least two candidate boxes, but is generated by the auditor of the second audit terminal. Therefore, the target terminal 70h can classify the at least two sample annotated images 70a by merging type, add the sample annotated images associated with the valid merging type to the first image set 70b, and add the sample annotated images associated with the invalid merging type to the second image set 70c. The target terminal 70h samples from the first image set 70b and the second image set 70c, and adds the collected sample annotated images to the training sample set 70d, as shown in FIG. Figure 7bAs shown, the probability of collecting sample annotated images in the first image set is 80%, that is, the first sampling probability, and the probability of collecting sample annotated images in the second image set is 20%, that is, the second sampling probability. Because the sample annotated images in the first image set are a combination of multi-person annotation results, while the sample annotated images in the second image set are single-person annotation results, the annotation boxes included in the first image set have higher accuracy. Therefore, when setting the sampling probability, the first sampling probability is greater than the second sampling probability. In this way, the training sample set will contain more sample annotated images associated with the effective merge type. Then, when training the initial image recognition model, the sample annotated images associated with the effective merge type have a high weight, which has a great impact on the initial image recognition model, thereby ensuring the accuracy of the adjusted generated image recognition model.

[0164] When the sample annotated image is collected, the training sample set 70d is determined, and the target terminal 70h inputs the training sample set 70d into the initial image recognition model. The following description is given by taking the annotated image and the original image as examples. The target terminal 70h determines the position information of the annotated box in the annotated image as the position label, and determines the object attribute category of the target object in the annotated image as the category label. The initial image recognition model extracts the image features of the original image. The target terminal 70h can obtain the predicted object attribute category and predicted position information of the target object based on the image features. It can be understood that the image features extracted by the initial image recognition model are incomplete at the beginning, so there will be errors in the predicted object attribute category and the category label, that is, the initial image recognition model has a large category loss value, and there will also be errors in the predicted position information and the position label, that is, the initial image recognition model has a large position loss value. The total loss value is determined based on the category loss value and the position loss value. The target terminal 70h needs to continuously adjust the weights of the parameters in the initial image recognition model until the total loss value converges, and the adjusted image recognition model is determined as the image recognition model.

[0165] In recent years, with the breakthrough of the new generation of artificial intelligence technology represented by deep learning, the field of automatic recognition of medical images has made great progress. The real-time detection and classification of medical images by artificial intelligence is expected to help clinicians improve the quality of examinations and reduce missed lesions. In medical images, the detection of endoscopic images is particularly difficult. Among the many artificial intelligence studies on digestive endoscopy, the intelligent detection or identification of colorectal polyps has received the most attention and the most in-depth research.

[0166] Excellent image recognition models rely on a large amount of representative high-quality annotated data, and the quality of data annotation determines the stability and accuracy of the algorithm model. However, due to the obvious individual differences and complexity of various types of endoscopic image lesions, the annotation difficulty is greater than that of general medical data. Effectively collecting large samples of high-quality annotated data has become the primary basis and prerequisite for the development of endoscopic images. In an embodiment of the present application, at least two candidate medical detection images are annotated by doctors with at least 3 years of endoscopic operation experience (and more than 1,000 endoscopic operations), the reviewer corresponding to the first audit terminal is a doctor with at least 5 years of endoscopic operation experience (and more than 3,000 endoscopic operations), and the reviewer corresponding to the second audit terminal is a doctor with at least 8 years of endoscopic operation experience (and more than 3,000 endoscopic operations). The accuracy of the annotation box in the annotated image is improved, and the accuracy of model recognition can be guaranteed.

[0167] Please see again Figure 7b The medical detection image 70h is an endoscopic image of the colorectum. The medical detection image 70e is input into the image recognition model. The image recognition model can obtain the medical image features of the medical detection image 70e. The target terminal 70h outputs the object detection category and the object position information (i.e., the center point position, width position and height position of the lesion in the medical detection image 70h) according to the medical image features, wherein the object detection category can be benign, malignant, a specific category in benign colorectum and a specific category in malignant colorectum, which is not limited here. A medical annotation box is generated according to the object position information, and finally the medical annotation box 70g and the object detection category are displayed on the display screen corresponding to the target terminal 70h, as shown in the medical detection image 70f.

[0168] In the embodiment of the present application, by acquiring at least two candidate images, at least two candidate frames that respectively carry the target objects for marking in the original image can be acquired, and the position information corresponding to the at least two candidate frames can be acquired. According to the position information corresponding to each two candidate frames, the overlapping position information between each two candidate frames corresponding thereto can be determined. According to the position information corresponding to each two candidate frames and the overlapping position information between each two candidate frames, the overlap degree between each two candidate frames can be determined. When the number of valid overlap degrees in the overlap degree is greater than a quantity threshold, it can be determined that the merging type corresponding to the at least two candidate frames is a valid merging type, and the candidate frame corresponding to the valid overlap degree is determined as At least two valid candidate frames; if the number of valid overlaps is less than the quantity threshold, the merge type corresponding to at least two candidate frames is determined to be an invalid merge type; if the merge type is a valid merge type, the original image carrying the merge candidate frame is sent to the first audit terminal corresponding to the valid merge type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merge candidate frame, and the above merge candidate frame can be determined according to at least two valid candidate frames; if the merge type is an invalid merge type, at least two candidate images are sent to the second audit terminal corresponding to the invalid merge type, so that the second audit terminal determines the annotated image corresponding to the original image according to at least two candidate images. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotation personnel by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotation frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, thereby improving the recognition accuracy of the model.

[0169] For further information, see Figure 8 , Figure 8 is a flow chart of an image processing method provided in an embodiment of the present application. Figure 2 As shown, the image processing process includes the following steps:

[0170] Step S201, obtaining at least two candidate images and annotation merging rules; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image; the annotation merging rule refers to a rule for determining the merging type based on the overlap between each two candidate boxes in the at least two candidate boxes.

[0171] Specifically, the number of at least two candidate images is 3; when there are 2 or 3 overlaps greater than the overlap threshold among the overlaps between every two candidate frames, the merge rule is marked to indicate that the merge type is a valid merge type; when there is 1 overlap greater than the overlap threshold among the overlaps between every two candidate frames, the merge rule is marked to indicate that the merge type is a valid merge type; when there is no overlap greater than the overlap threshold among the overlaps between every two candidate frames, the merge rule is marked to indicate that the merge type is an invalid merge type.

[0172] Please refer to Table 1, which is a pseudo code for the target terminal to implement the annotation merging rule when the number of at least two candidate images is 3.

[0173] Table 1

[0174]

[0175] Optionally, the number of at least two candidate images is 2; when the overlap between two candidate frames is greater than the overlap threshold, the annotation merge rule is used to indicate that the merge type is a valid merge type; when the overlap between two candidate frames is less than or equal to the overlap threshold, the annotation merge rule is used to indicate that the merge type is an invalid merge type.

[0176] Please refer to Table 2, which is a pseudo code for the target terminal to implement the annotation merging rule when the number of at least two candidate images is 2.

[0177] Table 2

[0178]

[0179] Please also see Fig. 9 , Fig. 9 It is a schematic diagram of an image processing scenario provided in an embodiment of the present application. Fig. 9 is an endoscopic image. Doctors 1, 2, and 3 are set to diagnose whether the endoscopic image contains the target object. If the target object is considered to exist, it is marked. If the target object is considered not to exist, the marking is abandoned. The pseudo code in Table 1 uses the intersection-over-union method to obtain the overlap between each two candidate boxes, where IOU is 12 express Fig. 9 The intersection-over-union ratio (i.e., overlap) between candidate box 1 and candidate box 2, IOU 13 express Fig. 9 The overlap between candidate box 1 and candidate box 3, IOU 23 express Fig. 9The degree of overlap between candidate box 3 and candidate box 2, 0.7 represents the threshold of the intersection-over-union ratio (i.e., the overlap threshold). If the overlap is greater than or equal to the overlap threshold, the target terminal can determine that the corresponding candidate box is a valid candidate box. If the overlap is less than the overlap threshold, the target terminal can determine that the corresponding candidate box is an invalid candidate box.

[0180] like Fig. 9 As shown, when three doctors have annotation results, if the three overlaps are all greater than 0.7, the target terminal determines that the merge type of candidate box 1, candidate box 2 and candidate box 3 is a valid merge type; if any two of the three overlaps are greater than 0.7, the target terminal determines that the merge type of candidate box 1, candidate box 2 and candidate box 3 is a valid merge type; if only one of the three overlaps is greater than 0.7, the target terminal determines that the merge type of candidate box 1, candidate box 2 and candidate box 3 is a valid merge type; if the three overlaps are all less than or equal to 0.7, the target terminal determines that the merge type of candidate box 1, candidate box 2 and candidate box 3 is an invalid merge type.

[0181] When both doctors have annotation results, if the overlap is greater than 0.7, the target terminal determines that the merging type of the two candidate boxes is a valid merging type; if the overlap is less than or equal to 0.7, the target terminal determines that the merging type of the two candidate boxes is an invalid merging type.

[0182] Step S202, merging at least two candidate boxes by marking the merging type indicated by the merging rule to obtain a review candidate result.

[0183] Specifically, by marking the valid merging type indicated by the merging rule, the three candidate boxes are merged to obtain a merged candidate box, and the audit candidate result is generated according to the merged candidate box.

[0184] By marking the valid merge type indicated by the merge rule, two candidate boxes whose overlap is greater than the overlap threshold are merged to obtain a merged candidate box, and an audit candidate result is generated based on the merged candidate box.

[0185] By marking the invalid merge types indicated by the merge rules, three candidate images are determined as review candidate results.

[0186] By marking the valid merge type indicated by the merge rule, two candidate boxes are merged to obtain a merged candidate box, and an audit candidate result is generated based on the merged candidate box.

[0187] By marking the invalid merge types indicated by the merge rule, the two candidate images are determined as the review candidate results.

[0188] See also Fig. 9, when three doctors have annotated results for the original image, that is, there are candidate boxes 1, 2 and 3. If there are 3 overlaps or 2 overlaps are greater than 0.7 among the 3 overlaps determined by the 3 candidate boxes, the merge type of the annotated image is a valid merge type. At this time, the target terminal generates the audit candidate result according to the average value of the 3 annotation results (that is, the position information corresponding to the 3 candidate boxes); if 1 overlap is greater than 0.7, the merge type is a valid merge type. At this time, the target terminal generates the audit candidate result according to the average value of the corresponding 2 annotation results. If IOU 13 Greater than 0.7, while IOU 12 and IOU 23 are both less than or equal to 0.7, the target terminal averages the position information of candidate box 1 and the position information of candidate box 3 to generate a candidate audit result; the above candidate audit results are all sent to the audit terminal (corresponding to Figure 2 In the first audit terminal in the embodiment, if the merge type is an invalid merge type, that is, the three overlaps are all less than or equal to 0.7, then the target terminal uses the three candidate images as audit candidate results, and the audit candidate results are sent to the arbitration terminal (corresponding to Figure 2 The second audit terminal in the embodiment).

[0189] In the case where both doctors have annotated results, assuming that the two annotated results are Fig. 9 Candidate box 1 and candidate box 2 in the image are used to determine the IOU based on candidate box 1 and candidate box 2. 12 , if IOU 12 If the value is greater than 0.7, the merge type is a valid merge type. At this time, the target terminal generates an audit candidate result based on the average value of the two annotation results, and the above audit candidate result is sent to the audit terminal; if the merge type is an invalid merge type, that is, IOU 12 Less than or equal to 0.7, at this time, the target terminal generates a review candidate result based on the three candidate images, and the review candidate result is sent to the arbitration terminal.

[0190] After the audit terminal obtains the audit candidate results, the auditor can confirm the audit candidate results. If the auditor approves the audit candidate results, the audit terminal can generate an annotation image based on the audit candidate results, such as Fig. 9 As shown, the annotated image includes an annotation box for annotating the target object in the original image; if the auditor does not approve the audit candidate result, the audit candidate result can be forwarded to the arbitration terminal so that the arbitration terminal determines the final audit candidate result.

[0191] Understandably, Figure 8 The valid merge types described in correspond to Figure 2 Valid merge types as described in Figure 8The invalid merge types described in Figure 2 Invalid merge type as described in .

[0192] This application takes into account the problem of excessive fluctuations in the annotation quality of a single candidate image, so the scenario where the number of candidate images is 1 is not considered.

[0193] In the embodiment of the present application, by obtaining at least two candidate images and annotation merging rules, the annotation merging rules refer to the rules for determining the merging type based on the overlap between each two candidate frames in the at least two candidate frames. First, at least two candidate frames that are respectively used to annotate the target object in the original image can be obtained, and the overlap between each two candidate frames in the at least two candidate frames can be determined. The merging type corresponding to the at least two candidate frames can be determined according to the overlap and the annotation merging rules. Then, the audit candidate results corresponding to the at least two candidate frames can be determined according to the merging type, and the audit candidate results can be sent to the audit terminal corresponding to the merging type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results, and the annotated frame in the annotated image includes the target object. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotator by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotated frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, and can improve the recognition accuracy of the model.

[0194] For further information, see Fig.10 , Fig.10 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The above-mentioned image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.10 As shown, the image processing device 1 may include: a first acquisition module 11 , a first determination module 12 , a second determination module 13 and a result sending module 14 .

[0195] A first acquisition module 11 is used to acquire at least two candidate images; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image;

[0196] A first determination module 12 is used to determine the overlap between every two candidate frames in at least two candidate frames;

[0197] A second determination module 13 is used to determine the merging type corresponding to at least two candidate boxes according to the overlap degree, and determine the candidate review results corresponding to at least two candidate boxes according to the merging type;

[0198] The sending result module 14 is used to send the audit candidate result to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result; the annotated box in the annotated image includes the target object.

[0199] The specific functional implementation of the first acquisition module 11, the first determination module 12, the second determination module 13 and the sending result module 14 can be referred to above. Figure 2 Steps S101 to S104 in the corresponding embodiment will not be described in detail here.

[0200] See also Fig.10 The second determining module 13 may include: a valid merging unit 131 and a first determining unit 132 .

[0201] The effective merging unit 131 is used for selecting at least two effective candidate frames from the at least two candidate frames if the merging type is an effective merging type, merging the at least two effective candidate frames to obtain a merged candidate frame, and generating a review candidate result according to the merged candidate frame;

[0202] The first determination unit 132 is used to determine at least two candidate images as review candidate results if the merge type is an invalid merge type.

[0203] The specific functional implementation of the effective merging unit 131 and the first determining unit 132 can be referred to above. Figure 2 The step S103 in the corresponding embodiment will not be described in detail here.

[0204] See also Fig.10 The second determination module 13 may include: a second determination unit 133 and a third determination unit 134 .

[0205] A second determining unit 133 is used to determine the overlap greater than the overlap threshold as a valid overlap, and determine the number of valid overlaps;

[0206] A third determining unit 134 is configured to determine that the merging type corresponding to at least two candidate frames is a valid merging type if the number of valid overlaps is equal to or greater than a number threshold;

[0207] The third determining unit 134 is further configured to determine that the merging type corresponding to at least two candidate frames is an invalid merging type if the number of valid overlaps is less than a number threshold.

[0208] The specific functional implementation of the second determining unit 133 and the third determining unit 134 can refer to the above Figure 2 The step S103 in the corresponding embodiment will not be described in detail here.

[0209] See also Fig.10 The effective merging unit 131 may include: a first determining subunit 1311 , an acquiring position subunit 1312 , a merging position subunit 1313 , and a generating candidate subunit 1314 .

[0210] A first determining subunit 1311 is used to determine candidate frames corresponding to the effective overlap degree as at least two effective candidate frames;

[0211] The position acquisition subunit 1312 is used to respectively acquire position information of at least two valid candidate frames in the corresponding candidate images;

[0212] The merged position subunit 1313 is used to merge at least two pieces of position information to obtain merged position information;

[0213] The candidate generation subunit 1314 is used to generate a merging candidate box indicated by the merging position information.

[0214] The specific functional implementation of the first determination subunit 1311, the acquisition position subunit 1312, the merging position subunit 1313 and the generation candidate subunit 1314 can be referred to above. Figure 2 The step S103 in the corresponding embodiment will not be described in detail here.

[0215] See also Fig.10 , the merge position subunit 1313 is specifically used to obtain the occurrence frequency of each valid candidate frame in the effective overlap degree, and determine the effective weight of each valid candidate frame according to the occurrence frequency;

[0216] The merged position subunit 1313 is further specifically configured to perform weighted averaging on at least two pieces of position information according to effective weights to obtain merged position information.

[0217] The specific functional implementation of the merge position subunit 1313 can be found in the above Figure 2 The step S103 in the corresponding embodiment will not be described in detail here.

[0218] See also Fig.10 The sending result module 14 is specifically used for sending the original image carrying the merging candidate frame to the first audit terminal corresponding to the valid merging type if the merging type is a valid merging type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merging candidate frame;

[0219] The sending result module 14 is also specifically used to send at least two candidate images to a second review terminal corresponding to the invalid merge type if the merge type is an invalid merge type, so that the second review terminal determines a labeled image corresponding to the original image based on the at least two candidate images.

[0220] The specific functional implementation of the sending result module 14 can be found in the above Figure 2 The step S104 in the corresponding embodiment will not be described in detail here.

[0221] See also Fig.10 The first determining module 12 may include: an information obtaining unit 121 , a fourth determining unit 122 , and a fifth determining unit 123 .

[0222] The information acquisition unit 121 is used to acquire the candidate frame K i In the candidate image P i Location information L i , get the candidate box K j In the candidate image P j Location information L j ;

[0223] The fourth determining unit 122 is configured to determine the position of the i And location information L j Determine the candidate box K i And the candidate box K j The overlap position information L ij ;

[0224] The fifth determining unit 123 is configured to determine the position of the i , location information j And the overlap position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0225] The specific functional implementation of the information acquisition unit 121, the fourth determination unit 122 and the fifth determination unit 123 can be referred to above. Figure 2 The step S102 in the corresponding embodiment will not be described in detail here.

[0226] See also Fig.10 The fifth determining unit 123 may include: a first acquiring subunit 1231 , a second acquiring subunit 1232 , a third acquiring subunit 1233 , and a coincidence determining subunit 1234 .

[0227] The first acquisition subunit 1231 is used to acquire the location information L i For the candidate box K i Height H i and width W i , according to the height H i and width W i Determine the candidate box Ki Area A i ;

[0228] The second acquisition subunit 1232 is used to acquire the location information L j For the candidate box K j Height H j and width W j , according to the height H j and width W j Determine the candidate box K j Area A j ;

[0229] The third acquisition subunit 1233 is used to obtain the overlap position information L ij In the example, for the candidate box K i And the candidate box K j The overlap height H ij And the overlap width W ij , according to the overlap height H ij And the overlap width W ij , determine the candidate box K i And the candidate box K j The overlap area A ij ;

[0230] Determine the overlap subunit 1234, which is used to determine the overlap subunit 1234 according to the area A i 、Area A j And the overlap area A ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0231] The specific functional implementation of the first acquisition subunit 1231, the second acquisition subunit 1232, the third acquisition subunit 1233 and the determination overlap subunit 1234 can be referred to above. Figure 2 The step S102 in the corresponding embodiment will not be described in detail here.

[0232] See also Fig.10 The image processing device 1 may further include: a second acquisition module 15 and a third acquisition module 16 .

[0233] The second acquisition module 15 is used to acquire a training sample set; the training sample set includes annotated images and original images;

[0234] The second acquisition module is further used to determine the position information of the annotation box in the annotated image as the position label, and determine the object attribute category of the target object in the annotated image as the category label;

[0235] The second acquisition module 15 is also used to input the original image into the initial image recognition model;

[0236] A third acquisition module 16 is used to acquire image features of the original image according to the initial image recognition model, and predict the predicted object attribute category and predicted position information of the target object according to the image features;

[0237] The third acquisition module 16 is further used to adjust the initial image recognition model according to the predicted object attribute category, predicted location information, category label and location label to obtain the image recognition model.

[0238] The specific functional implementation of the second acquisition module 15 and the third acquisition module 16 can be found in the above Figure 2 The step S104 in the corresponding embodiment will not be described in detail here.

[0239] See also Fig.10 The third acquisition module 16 may include: a sixth determination unit 161 and a seventh determination unit 162 .

[0240] A sixth determining unit 161, configured to determine a category loss value according to the predicted object attribute category and the category label;

[0241] The sixth determining unit 161 is further configured to determine a position loss value according to the predicted position information and the position tag;

[0242] The sixth determining unit 161 is further used to determine the total loss value according to the category loss value and the position loss value;

[0243] The seventh determination unit 162 is used to adjust the weights of the parameters in the initial image recognition model according to the total loss value, and when the total loss value meets the convergence condition, determine the adjusted image recognition model as the image recognition model.

[0244] The specific functional implementation of the sixth determining unit 161 and the seventh determining unit 162 can refer to the above Figure 2 The step S104 in the corresponding embodiment will not be described in detail here.

[0245] See also Fig.10 The image processing device 1 may further include: a fourth acquisition module 17.

[0246] A fourth acquisition module 17 is used to acquire at least two sample annotated images; the at least two sample annotated images include an annotated image;

[0247] The fourth acquisition module 17 is further used to add, among the at least two sample annotated images, the sample annotated images associated with the valid merge type to the first image set, and add the sample annotated images associated with the invalid merge type to the second image set; the first image set has a first sampling probability, the second image set has a second sampling probability, and the first sampling probability is greater than the second sampling probability;

[0248] The fourth acquisition module 17 is further used to collect sample annotated images between the first image set and the second image set according to the first sampling probability and the second sampling probability, and add the collected sample annotated images to the training sample set.

[0249] The specific functional implementation of the fourth acquisition module 17 can be found in the above Figure 2 The step S104 in the corresponding embodiment will not be described in detail here.

[0250] See also Fig.10 The image processing device 1 may further include: a fifth acquisition module 18

[0251] The fifth acquisition module 18 is used to acquire a medical detection image; the medical detection image includes an object to be identified;

[0252] The fifth acquisition module 18 is further used to input the medical detection image into the image recognition model, and obtain the medical image features of the medical detection image in the image recognition model;

[0253] The fifth acquisition module 18 is also used to output the object detection category and the object position information according to the medical image features in the image recognition model;

[0254] The fifth acquisition module 18 is further used to generate a medical annotation frame according to the object position information; the medical annotation frame is used to mark the object to be identified in the medical detection image;

[0255] The fifth acquisition module 18 is further used to display the medical annotation box and the object detection category in the medical detection image.

[0256] The specific functional implementation of the fifth acquisition module 18 can be found in the above Figure 2 The step S104 in the corresponding embodiment will not be described in detail here.

[0257] In the embodiment of the present application, by acquiring at least two candidate images, at least two candidate frames that respectively carry the target objects for marking in the original image can be acquired, and the position information corresponding to the at least two candidate frames can be acquired. According to the position information corresponding to each two candidate frames, the overlapping position information between each two candidate frames corresponding thereto can be determined. According to the position information corresponding to each two candidate frames and the overlapping position information between each two candidate frames, the overlap degree between each two candidate frames can be determined. When the number of valid overlap degrees in the overlap degree is greater than a quantity threshold, it can be determined that the merging type corresponding to the at least two candidate frames is a valid merging type, and the candidate frame corresponding to the valid overlap degree is determined as At least two valid candidate frames; if the number of valid overlaps is less than the quantity threshold, the merge type corresponding to at least two candidate frames is determined to be an invalid merge type; if the merge type is a valid merge type, the original image carrying the merge candidate frame is sent to the first audit terminal corresponding to the valid merge type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merge candidate frame, and the above merge candidate frame can be determined according to at least two valid candidate frames; if the merge type is an invalid merge type, at least two candidate images are sent to the second audit terminal corresponding to the invalid merge type, so that the second audit terminal determines the annotated image corresponding to the original image according to at least two candidate images. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotation personnel by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotation frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, thereby improving the recognition accuracy of the model.

[0258] For further information, see Fig.11 , Fig.11 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The above-mentioned image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.11 As shown, the image processing device 2 may include: a rule acquisition module 21 and a candidate merging module 22 .

[0259] In an exemplary embodiment, the image processing apparatus may be used to perform Figure 8 The corresponding steps in the method shown; then:

[0260] The acquisition rule module 21 is used to acquire at least two candidate images and an annotation merging rule; the at least two candidate images respectively carry candidate frames for annotating the target object in the original image; the annotation merging rule refers to a rule for determining a merging type based on the overlap between each two candidate frames in the at least two candidate frames;

[0261] The candidate merging module 22 is used to merge at least two candidate boxes by marking the merging type indicated by the merging rule to obtain the audit candidate result.

[0262] The specific functional implementation of the rule acquisition module 21 and the candidate merging module 22 can be found in the above Figure 8 The steps S201 and S202 in the corresponding embodiment will not be described in detail here.

[0263] In one implementation, the number of at least two candidate images is 3; when there are 2 or 3 overlaps between every two candidate frames that are greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0264] The merging candidate module 22 is specifically used to merge the three candidate boxes by marking the valid merging type indicated by the merging rule to obtain a merged candidate box, and generate a review candidate result according to the merged candidate box.

[0265] The specific functional implementation of the merge candidate module 22 can be found in the above Figure 8 The step S202 in the corresponding embodiment will not be described in detail here.

[0266] In one implementation, the number of at least two candidate images is 3; when one of the overlaps between every two candidate frames is greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0267] The merging candidate module 22 is specifically used to merge two candidate boxes whose overlap is greater than an overlap threshold by marking the valid merging type indicated by the merging rule, obtain a merged candidate box, and generate an audit candidate result based on the merged candidate box.

[0268] The specific functional implementation of the merge candidate module 22 can be found in the above Figure 8 The step S202 in the corresponding embodiment will not be described in detail here.

[0269] In one implementation, the number of the at least two candidate images is 3; when there is no overlap greater than the overlap threshold value in the overlap between each two candidate frames, the merging rule is marked to indicate that the merging type is an invalid merging type;

[0270] The merging candidate module 22 is specifically used to determine the three candidate images as the audit candidate results by marking the invalid merging types indicated by the merging rules.

[0271] The specific functional implementation of the merge candidate module 22 can be found in the above Figure 8 The step S202 in the corresponding embodiment will not be described in detail here.

[0272] In one implementation, the number of the at least two candidate images is 2; when the overlap between the two candidate frames is greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0273] The merging candidate module 22 is specifically used to merge two candidate boxes by marking the valid merging type indicated by the merging rule to obtain a merged candidate box, and generate a review candidate result according to the merged candidate box.

[0274] The specific functional implementation of the merge candidate module 22 can be found in the above Figure 8 The step S202 in the corresponding embodiment will not be described in detail here.

[0275] In one implementation, the number of the at least two candidate images is 2; when the overlap between the two candidate frames is less than or equal to the overlap threshold, the marking merge rule is used to indicate that the merge type is an invalid merge type;

[0276] The merging candidate module 22 is specifically used to determine two candidate images as review candidate results by marking the invalid merging types indicated by the merging rules.

[0277] In the embodiment of the present application, by obtaining at least two candidate images and annotation merging rules, the annotation merging rules refer to the rules for determining the merging type based on the overlap between each two candidate frames in the at least two candidate frames. First, at least two candidate frames that are respectively used to annotate the target object in the original image can be obtained, and the overlap between each two candidate frames in the at least two candidate frames can be determined. The merging type corresponding to the at least two candidate frames can be determined according to the overlap and the annotation merging rules. Then, the audit candidate results corresponding to the at least two candidate frames can be determined according to the merging type, and the audit candidate results can be sent to the audit terminal corresponding to the merging type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results, and the annotated frame in the annotated image includes the target object. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotator by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotated frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, and can improve the recognition accuracy of the model.

[0278] For further information, see Fig.12 , Fig.12 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.12 As shown, the above-mentioned computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.12 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0279] exist Fig.12 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0280] Acquire at least two candidate images; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image;

[0281] Determine the overlap between every two candidate frames in at least two candidate frames;

[0282] Determine a merging type corresponding to at least two candidate boxes according to the degree of overlap, and determine candidate review results corresponding to at least two candidate boxes according to the merging type;

[0283] The audit candidate result is sent to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result; the annotated box in the annotated image includes the target object.

[0284] In one embodiment, when the processor 1001 determines the review candidate results corresponding to at least two candidate boxes according to the merge type, the processor 1001 specifically performs the following steps:

[0285] If the merge type is a valid merge type, selecting at least two valid candidate boxes from the at least two candidate boxes, merging the at least two valid candidate boxes to obtain a merged candidate box, and generating a review candidate result according to the merged candidate box;

[0286] If the merge type is an invalid merge type, at least two candidate images are respectively determined as review candidate results.

[0287] In one embodiment, when the processor 1001 determines the merging type corresponding to at least two candidate frames according to the overlap degree, the processor 1001 specifically performs the following steps:

[0288] Determine the overlap degree greater than the overlap degree threshold as the effective overlap degree, and determine the number of effective overlap degrees;

[0289] If the number of valid overlaps is equal to or greater than the number threshold, determining that the merging type corresponding to at least two candidate frames is a valid merging type;

[0290] If the number of valid overlaps is less than the number threshold, it is determined that the merging type corresponding to at least two candidate frames is an invalid merging type.

[0291] In one embodiment, when the processor 1001 selects at least two valid candidate boxes from at least two candidate boxes and merges the at least two valid candidate boxes to obtain a merged candidate box, the processor 1001 specifically performs the following steps:

[0292] Determine candidate frames corresponding to the effective overlap as at least two effective candidate frames;

[0293] Respectively obtain position information of at least two valid candidate frames in the corresponding candidate images;

[0294] Merging at least two pieces of position information to obtain merged position information;

[0295] Generate a merging candidate box indicated by the merging position information.

[0296] In one embodiment, when the processor 1001 merges at least two pieces of position information to obtain merged position information, the processor 1001 specifically performs the following steps:

[0297] Obtain the frequency of occurrence of each valid candidate box in the effective overlap, and determine the effective weight of each valid candidate box according to the frequency of occurrence;

[0298] A weighted average is performed on at least two pieces of position information according to effective weights to obtain combined position information.

[0299] In one embodiment, the audit terminal includes a first audit terminal and a second audit terminal;

[0300] When the processor 1001 sends the audit candidate result to the audit terminal corresponding to the merge type so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result, the processor 1001 specifically performs the following steps:

[0301] If the merge type is a valid merge type, the original image carrying the merge candidate frame is sent to a first audit terminal corresponding to the valid merge type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merge candidate frame;

[0302] If the merge type is an invalid merge type, at least two candidate images are sent to a second audit terminal corresponding to the invalid merge type, so that the second audit terminal determines a labeled image corresponding to the original image based on the at least two candidate images.

[0303] In one embodiment, the at least two candidate images include candidate image P i And the candidate image P j , candidate image P i Carry candidate box W i , candidate image P j Carry candidate box W j , where i and j are positive integers, and i and j are less than or equal to the total number of candidate images in at least two candidate images;

[0304] When determining the overlap between every two candidate frames in at least two candidate frames, the processor 1001 specifically performs the following steps:

[0305] Get the candidate box K i In the candidate image P i Location information L i , get the candidate box K j In the candidate image P j Location information L j ;

[0306] According to the location information L i And location information L j Determine the candidate box K i And the candidate box K j The overlap position information L ij ;

[0307] According to the location information L i , location information j And the overlap position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0308] In one embodiment, the processor 1001 executes the i , location information j And the overlap position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij When the system is running, perform the following steps:

[0309] Get location information i For the candidate box K i Height H i and width W i , according to the height H i and width W i Determine the candidate box K i Area A i ;

[0310] Get location information j For the candidate box K j Height H j and width W j , according to the height H j and width W j Determine the candidate box K j Area A j ;

[0311] Get the overlap position information L ij In the example, for the candidate box K i And the candidate box K j The overlap height H ij And the overlap width W ij , according to the overlap height H ij And the overlap width W ij , determine the candidate box K i And the candidate box K j The overlap area A ij ;

[0312] According to the area A i 、Area A j And the overlap area A ij , determine the candidate box K i And the candidate box K j The overlap between ij .

[0313] In one embodiment, the processor 1001 further specifically performs the following steps:

[0314] Obtain a training sample set; the training sample set includes annotated images and original images;

[0315] Determine the position information of the annotation box in the annotated image as the position label, and determine the object attribute category of the target object in the annotated image as the category label;

[0316] Input the original image into the initial image recognition model;

[0317] Acquire image features of the original image according to the initial image recognition model, and predict the predicted object attribute category and predicted position information of the target object according to the image features;

[0318] According to the predicted object attribute category, predicted location information, category label and location label, the initial image recognition model is adjusted to obtain an image recognition model.

[0319] In one embodiment, when the processor 1001 adjusts the initial image recognition model according to the predicted object attribute category, predicted location information, category label, and location label to obtain the image recognition model, the following steps are specifically performed:

[0320] Determine the category loss value based on the predicted object attribute category and category label;

[0321] Determine the position loss value according to the predicted position information and the position label;

[0322] Determine a total loss value based on the category loss value and the location loss value;

[0323] The weights of the parameters in the initial image recognition model are adjusted according to the total loss value, and when the total loss value meets the convergence condition, the adjusted image recognition model is determined as the image recognition model.

[0324] In one embodiment, the processor 1001 further specifically performs the following steps:

[0325] Acquire at least two sample annotated images; the at least two sample annotated images include an annotated image;

[0326] Among at least two sample annotated images, the sample annotated image associated with the valid merge type is added to a first image set, and the sample annotated image associated with the invalid merge type is added to a second image set; the first image set has a first sampling probability, the second image set has a second sampling probability, and the first sampling probability is greater than the second sampling probability;

[0327] According to the first sampling probability and the second sampling probability, sample annotated images are collected between the first image set and the second image set, and the collected sample annotated images are added to the training sample set.

[0328] In one embodiment, the processor 1001 further specifically performs the following steps:

[0329] Acquire a medical detection image; the medical detection image includes an object to be identified;

[0330] Inputting the medical detection image into the image recognition model, and obtaining the medical image features of the medical detection image in the image recognition model;

[0331] In the image recognition model, the object detection category and object location information are output based on the medical image features;

[0332] Generate a medical annotation frame according to the object position information; the medical annotation frame is used to mark the object to be identified in the medical detection image;

[0333] In medical detection images, medical annotation boxes and object detection categories are displayed.

[0334] In the embodiment of the present application, by acquiring at least two candidate images, at least two candidate frames that respectively carry the target objects for marking in the original image can be acquired, and the position information corresponding to the at least two candidate frames can be acquired. According to the position information corresponding to each two candidate frames, the overlapping position information between each two candidate frames corresponding thereto can be determined. According to the position information corresponding to each two candidate frames and the overlapping position information between each two candidate frames, the overlap degree between each two candidate frames can be determined. When the number of valid overlap degrees in the overlap degree is greater than a quantity threshold, it can be determined that the merging type corresponding to the at least two candidate frames is a valid merging type, and the candidate frame corresponding to the valid overlap degree is determined as At least two valid candidate frames; if the number of valid overlaps is less than the quantity threshold, the merge type corresponding to at least two candidate frames is determined to be an invalid merge type; if the merge type is a valid merge type, the original image carrying the merge candidate frame is sent to the first audit terminal corresponding to the valid merge type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merge candidate frame, and the above merge candidate frame can be determined according to at least two valid candidate frames; if the merge type is an invalid merge type, at least two candidate images are sent to the second audit terminal corresponding to the invalid merge type, so that the second audit terminal determines the annotated image corresponding to the original image according to at least two candidate images. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotation personnel by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotation frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, thereby improving the recognition accuracy of the model.

[0335] See also Fig.13 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13As shown, the above-mentioned computer device 2000 may include: a processor 2001, a network interface 2004 and a memory 2005. In addition, the above-mentioned computer device 2000 may also include: a user interface 2003, and at least one communication bus 2002. Among them, the communication bus 2002 is used to realize the connection and communication between these components. Among them, the user interface 2003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 2003 may also include a standard wired interface and a wireless interface. The network interface 2004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 2005 may optionally also be at least one storage device located away from the aforementioned processor 2001. As Fig.13 As shown, the memory 2005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0336] exist Fig.13 In the computer device 2000 shown, the network interface 2004 can provide a network communication function; the user interface 2003 is mainly used to provide an input interface for the user; and the processor 2001 can be used to call the device control application stored in the memory 2005 to achieve:

[0337] Acquire at least two candidate images and a labeling merging rule; the at least two candidate images respectively carry candidate frames for labeling target objects in the original image; the labeling merging rule refers to a rule for determining a merging type based on a degree of overlap between each two candidate frames in the at least two candidate frames;

[0338] By marking the merging type indicated by the merging rule, at least two candidate boxes are merged to obtain a review candidate result.

[0339] In one embodiment, the number of at least two candidate images is 3; when 2 or 3 of the overlaps between every two candidate frames are greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0340] When the processor 2001 executes the merging type indicated by the marked merging rule and merges at least two candidate boxes to obtain the audit candidate result, the processor 2001 specifically performs the following steps:

[0341] By marking the valid merge type indicated by the merge rule, the three candidate boxes are merged to obtain a merged candidate box, and the review candidate result is generated according to the merged candidate box.

[0342] In one embodiment, the number of at least two candidate images is 3; when one of the overlaps between every two candidate frames is greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0343] When the processor 2001 executes the merging type indicated by the marked merging rule and merges at least two candidate boxes to obtain the audit candidate result, the processor 2001 specifically performs the following steps:

[0344] By marking the valid merge type indicated by the merge rule, two candidate boxes whose overlap is greater than the overlap threshold are merged to obtain a merged candidate box, and an audit candidate result is generated based on the merged candidate box.

[0345] In one embodiment, the number of the at least two candidate images is 3; when there is no overlap greater than the overlap threshold value in the overlap between each two candidate frames, the merging rule is marked to indicate that the merging type is an invalid merging type;

[0346] When the processor 2001 executes the merging type indicated by the marked merging rule and merges at least two candidate boxes to obtain the audit candidate result, the processor 2001 specifically performs the following steps:

[0347] By marking the invalid merge types indicated by the merge rules, three candidate images are determined as review candidate results.

[0348] In one embodiment, the number of the at least two candidate images is 2; when the overlap between the two candidate frames is greater than the overlap threshold, the merging rule is marked to indicate that the merging type is a valid merging type;

[0349] When the processor 2001 executes the merging type indicated by the marked merging rule and merges at least two candidate boxes to obtain the audit candidate result, the processor 2001 specifically performs the following steps:

[0350] By marking the valid merge type indicated by the merge rule, two candidate boxes are merged to obtain a merged candidate box, and an audit candidate result is generated based on the merged candidate box.

[0351] In one embodiment, the number of the at least two candidate images is 2; when the overlap between the two candidate frames is less than or equal to the overlap threshold, the merging rule is marked to indicate that the merging type is an invalid merging type;

[0352] When the processor 2001 executes the merging type indicated by the marked merging rule and merges at least two candidate boxes to obtain the audit candidate result, the processor 2001 specifically performs the following steps:

[0353] By marking the invalid merge types indicated by the merge rule, the two candidate images are determined as the review candidate results.

[0354] In the embodiment of the present application, by obtaining at least two candidate images and annotation merging rules, the annotation merging rules refer to the rules for determining the merging type based on the overlap between each two candidate frames in the at least two candidate frames. First, at least two candidate frames that are respectively used to annotate the target object in the original image can be obtained, and the overlap between each two candidate frames in the at least two candidate frames can be determined. The merging type corresponding to the at least two candidate frames can be determined according to the overlap and the annotation merging rules. Then, the audit candidate results corresponding to the at least two candidate frames can be determined according to the merging type, and the audit candidate results can be sent to the audit terminal corresponding to the merging type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate results, and the annotated frame in the annotated image includes the target object. As can be seen from the above, the present application can make up for the problem of insufficient individual ability or ability difference of a single annotator by cross-annotating and merging at least two candidate images, and can improve the accuracy of the annotated frame for the target object in the annotated image. When the annotated image with high accuracy is input into the algorithm model as a training sample, it can promote the algorithm model to better learn the target features in the training sample, and can improve the recognition accuracy of the model.

[0355] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figure 2 and Figure 8 For details on the image processing methods provided in each step, please refer to the above Figure 2 and Figure 8 The implementation methods provided by each step will not be repeated here.

[0356] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0357] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0358] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0359] The method and related device provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application, and can be specifically implemented by computer program instructions for each process and / or box of the method flow chart and / or structural diagram, and the combination of the process and / or box in the flow chart and / or block diagram. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the structure diagram.

[0360] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Obtain at least two candidate images; The at least two candidate images respectively carry candidate boxes for marking the target object in the original image; Determine the overlap between every two candidate frames in at least two candidate frames; Determine the overlap degree greater than the overlap degree threshold as the effective overlap degree, and determine the number of the effective overlap degrees; If the number of the effective overlaps is equal to or greater than the number threshold, determining the effective merge type as the merge type corresponding to the at least two candidate frames; The effective merging type is used to indicate to merge the candidate frames corresponding to the effective overlap degree; If the number of the valid overlaps is less than the number threshold, the invalid merging type is used to determine the merging type corresponding to the at least two candidate frames; The invalid merge type is used to indicate that the at least two candidate images are respectively determined as review candidate results; Determine, according to the merging type, candidate review results corresponding to the at least two candidate boxes; The audit candidate result determined based on the effective merge type is a merge candidate frame, the merge candidate frame is determined based on merge position information, the merge position information is obtained by weighted averaging position information of at least two effective candidate frames according to effective weights of the at least two effective candidate frames, the at least two effective candidate frames are candidate frames corresponding to the effective overlap, and the effective weight of each effective candidate frame is determined according to the appearance frequency of each effective candidate frame in the effective overlap; Sending the audit candidate result to the audit terminal corresponding to the merge type, so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result; The annotation box in the annotated image includes the target object.

2. The method according to claim 1, characterized in that The determining, according to the merging type, candidate review results corresponding to the at least two candidate boxes includes: If the merging type is a valid merging type, selecting at least two valid candidate frames from the at least two candidate frames, merging the at least two valid candidate frames to obtain a merged candidate frame, and generating the audit candidate result according to the merged candidate frame; If the merge type is an invalid merge type, the at least two candidate images are respectively determined as the review candidate results.

3. The method according to claim 2, characterized in that The selecting at least two valid candidate frames from the at least two candidate frames, and merging the at least two valid candidate frames to obtain a merged candidate frame, includes: Determining the candidate frames corresponding to the effective overlap degree as the at least two effective candidate frames; Respectively obtaining position information of the at least two valid candidate frames in the corresponding candidate images; Merging at least two pieces of position information to obtain the merged position information; The merging candidate frame indicated by the merging position information is generated.

4. The method according to claim 3, characterized in that The step of merging at least two pieces of position information to obtain the merged position information includes: Obtaining the occurrence frequency of each valid candidate frame in the effective overlap degree, and determining the effective weight of each valid candidate frame according to the occurrence frequency; The at least two pieces of position information are weighted averaged according to the effective weight to obtain the combined position information.

5. The method according to claim 1, characterized in that The audit terminal includes a first audit terminal and a second audit terminal; The step of sending the audit candidate result to the audit terminal corresponding to the merge type so that the audit terminal determines the annotated image corresponding to the original image according to the audit candidate result includes: If the merging type is the valid merging type, the original image carrying the merging candidate frame is sent to the first audit terminal corresponding to the valid merging type, so that the first audit terminal determines the annotated image corresponding to the original image according to the merging candidate frame; If the merge type is the invalid merge type, the at least two candidate images are sent to the second audit terminal corresponding to the invalid merge type, so that the second audit terminal determines the annotated image corresponding to the original image based on the at least two candidate images.

6. The method according to claim 1, characterized in that The at least two candidate images include candidate image P i And the candidate image P j , the candidate image P i Carry candidate box W i , the candidate image P j Carry candidate box W j , wherein i and j are positive integers, and i and j are less than or equal to the total number of candidate images in the at least two candidate images; The determining the overlap between every two candidate frames in at least two candidate frames comprises: Get the candidate box K i In the candidate image P i Location information in L i , get the candidate box K j In the candidate image P j Location information in L j ; According to the location information L i And the location information L j Determine the candidate box K i And the candidate box K j The overlap position information L ij ; According to the location information L i , the location information L j And the overlap position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij .

7. The method according to claim 6, characterized in that The position information L i , the location information L j And the overlap position information L ij , determine the candidate box K i And the candidate box K j The overlap between ij ,include: Get the location information L i For the candidate box K i Height H i and width W i , according to the height H i and the width W i Determine the candidate box K i Area A i ; Get the location information L j For the candidate box K j Height H j and width W j , according to the height H j and the width W j Determine the candidate box K j Area A j ; Get the overlap position information L ij In the example, for the candidate box K i And the candidate box K j The overlap height H ij And the overlap width W ij , according to the overlap height H ij And the overlap width W ij , determine the candidate box K i And the candidate box K j The overlap area A ij ; According to the area A i The area A j And the overlapping area A ij , determine the candidate box K i And the candidate box K j The overlap between ij .

8. The method according to claim 2, characterized in that: Also includes: Acquire a training sample set; the training sample set includes the annotated image and the original image; Determine the position information of the annotation box in the annotated image as a position label, and determine the object attribute category of the target object in the annotated image as a category label; Inputting the original image into an initial image recognition model; Acquire image features of the original image according to the initial image recognition model, and predict the predicted object attribute category and predicted position information of the target object according to the image features; The initial image recognition model is adjusted according to the predicted object attribute category, the predicted position information, the category label and the position label to obtain an image recognition model.

9. The method according to claim 8, characterized in that The adjusting the initial image recognition model according to the predicted object attribute category, the predicted location information, the category label and the location label to obtain the image recognition model includes: Determine a category loss value according to the predicted object attribute category and the category label; Determine a position loss value according to the predicted position information and the position tag; Determine a total loss value according to the category loss value and the position loss value; The weights of the parameters in the initial image recognition model are adjusted according to the total loss value, and when the total loss value meets the convergence condition, the adjusted image recognition model is determined as the image recognition model.

10. The method according to claim 8, characterized in that Also includes: Acquire at least two sample annotated images; the at least two sample annotated images include the annotated image; Among the at least two sample annotated images, the sample annotated image associated with the valid merge type is added to a first image set, and the sample annotated image associated with the invalid merge type is added to a second image set; the first image set has a first sampling probability, the second image set has a second sampling probability, and the first sampling probability is greater than the second sampling probability; According to the first sampling probability and the second sampling probability, sample annotated images are collected between the first image set and the second image set, and the collected sample annotated images are added to the training sample set.

11. The method according to claim 8, characterized in that Also includes: Acquire a medical detection image; the medical detection image includes an object to be identified; Inputting the medical detection image into the image recognition model, and obtaining medical image features of the medical detection image in the image recognition model; In the image recognition model, outputting object detection categories and object location information according to the medical image features; generating a medical annotation frame according to the object position information; The medical annotation frame is used to mark the object to be identified in the medical detection image; In the medical detection image, the medical annotation box and the object detection category are displayed.

12. An image processing method, characterized in that: include: Acquire at least two candidate images and a marking merging rule; the at least two candidate images respectively carry candidate boxes for marking target objects in the original image; The annotation merging rule refers to a rule for determining a merging type based on the overlap between every two candidate frames in at least two candidate frames; the merging type includes a valid merging type and an invalid merging type; The effective merge type refers to a merge type in which the number of effective overlaps is equal to or greater than the number threshold value; The valid merge type is used to indicate that the candidate frames corresponding to the valid overlap are merged; the invalid merge type refers to the merge type indicated by the number of valid overlaps being less than the number threshold; the valid overlap includes the overlap between each two candidate frames in the at least two candidate frames that is greater than the overlap threshold; the invalid merge type is used to indicate that the at least two candidate images are respectively determined as the audit candidate results; Merge the at least two candidate boxes according to the merge type indicated by the marked merge rule to obtain a review candidate result; The review candidate result determined based on the valid merge type is a merge candidate box, and the merge candidate box is determined based on merge position information, and the merge position information is obtained by weighted averaging the position information of the at least two valid candidate boxes according to their effective weights, and the at least two valid candidate boxes are candidate boxes corresponding to the effective overlap, and the effective weight of each valid candidate box is determined according to the frequency of occurrence of each valid candidate box in the effective overlap.

13. The method according to claim 12, characterized in that The number of the at least two candidate images is 3; when there are 2 or 3 overlaps between every two candidate frames that are greater than the overlap threshold, the marking merge rule is used to indicate that the merge type is a valid merge type; The step of merging the at least two candidate boxes according to the merging type indicated by the marking merging rule to obtain a review candidate result includes: The three candidate boxes are merged according to the valid merge type indicated by the marked merge rule to obtain a merged candidate box, and the audit candidate result is generated according to the merged candidate box.

14. The method according to claim 12, characterized in that The number of the at least two candidate images is 3; when one of the overlaps between every two candidate frames is greater than the overlap threshold, the marking merge rule is used to indicate that the merge type is a valid merge type; The step of merging the at least two candidate boxes according to the merging type indicated by the marking merging rule to obtain a review candidate result includes: Using the valid merge type indicated by the marked merge rule, two candidate boxes whose overlap is greater than the overlap threshold are merged to obtain a merged candidate box, and the audit candidate result is generated based on the merged candidate box.

15. The method according to claim 12, characterized in that The number of the at least two candidate images is 3; when there is no overlap greater than the overlap threshold value in the overlap between each two candidate frames, the marking merge rule is used to indicate that the merge type is an invalid merge type; The step of merging the at least two candidate boxes according to the merging type indicated by the marking merging rule to obtain a review candidate result includes: By using the invalid merge type indicated by the marked merge rule, three candidate images are respectively determined as the review candidate results.

16. The method according to claim 12, characterized in that The number of the at least two candidate images is 2; when the overlap between the two candidate frames is greater than the overlap threshold, the marking merge rule is used to indicate that the merge type is a valid merge type; The step of merging the at least two candidate boxes according to the merging type indicated by the marking merging rule to obtain a review candidate result includes: The two candidate boxes are merged according to the valid merge type indicated by the marked merge rule to obtain a merged candidate box, and the audit candidate result is generated according to the merged candidate box.

17. The method according to claim 12, characterized in that The number of the at least two candidate images is 2; when the overlap between the two candidate frames is less than or equal to the overlap threshold, the marking merge rule is used to indicate that the merge type is an invalid merge type; The step of merging the at least two candidate boxes according to the merging type indicated by the marking merging rule to obtain a review candidate result includes: By using the invalid merge type indicated by the marked merge rule, two candidate images are determined as the review candidate results.

18. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code to execute the steps of the method described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 17 are executed.