Image data processing method, computer device and medium

Abstract

The embodiment of the present application provides an image data processing method, a computer device and a medium, the method comprising: obtaining an initial network model from a server, annotating an original image based on the initial network model, and obtaining a predicted annotation area for annotating abnormal objects in the original image; obtaining a candidate annotation area provided by the annotated object for annotating abnormal objects in the original image, performing an area review on the candidate annotation area, and obtaining a standard annotation area for annotating abnormal objects; adjusting the model parameters of the initial network model based on the predicted annotation area and the standard annotation area, obtaining a first updated model parameter, and sending the first updated model parameter to the server; receiving a target model parameter returned by the server, updating the initial network model by the target model parameter, and obtaining a target network model for annotating abnormal objects. By adopting the present application, the accuracy of auxiliary annotation by the network model can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an image data processing method, computer equipment and medium. Background Art

[0002] In the existing data labeling scheme, the annotator can use the artificial intelligence model to predict the data to be labeled and obtain the pre-labeling results corresponding to the data to be labeled. In this way, the annotator can complete the data labeling of the data to be labeled in combination with the pre-labeling results and obtain the final labeling results corresponding to the data to be labeled. Among them, the annotator cannot determine the difference between the pre-labeling results and the final labeling results, and the annotator's labeling work cannot affect the labeling effect of the artificial intelligence model.

[0003] In addition, the AI ​​models currently in use are all based on existing pre-trained models, and the update of the pre-trained models depends on the algorithm personnel. The algorithm personnel can obtain the data required for the model update from the outside, and then use the centralized data to train the model. On the one hand, since data acquisition is slow, the data-driven model update is also slow, resulting in the inability to update the model in a timely manner. On the other hand, the breadth of data collected by algorithm personnel is very limited, resulting in the inability of existing AI models to predict the more extensive data to be labeled, which in turn limits the accuracy of AI models in assisting labeling. Summary of the invention

[0004] The embodiments of the present application provide an image data processing method, a computer device, and a medium, which can improve the accuracy of auxiliary annotation of a network model.

[0005] An embodiment of the present application provides an image data processing method, including:

[0006] The first annotation terminal obtains an initial network model from the server, and annotates the original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the original image;

[0007] Acquire a candidate annotation region provided by the annotation object for annotating an abnormal object in the original image, perform a region review on the candidate annotation region, and obtain a standard annotation region for annotating the abnormal object;

[0008] Based on the predicted annotation area and the standard annotation area, the model parameters of the initial network model are adjusted to obtain first updated model parameters, and the first updated model parameters are sent to the server so that the server performs parameter fusion on the first updated model parameters and the second updated model parameters to generate target model parameters; the second updated model parameters are provided by the second annotation terminal;

[0009] The target model parameters returned by the server are received, and the initial network model is updated by the target model parameters to obtain a target network model for marking regions of abnormal objects.

[0010] An embodiment of the present application provides an image data processing device, including:

[0011] A model acquisition module, used for the first annotation terminal to obtain an initial network model from a server, and to annotate the original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the original image;

[0012] A region review module is used to obtain a candidate marked region provided by the marked object for marking an abnormal object in the original image, perform a region review on the candidate marked region, and obtain a standard marked region for marking the abnormal object;

[0013] A first sending module is used to adjust the model parameters of the initial network model based on the predicted annotation area and the standard annotation area to obtain the first updated model parameters, and send the first updated model parameters to the server so that the server performs parameter fusion on the first updated model parameters and the second updated model parameters to generate the target model parameters; the second updated model parameters are provided by the second annotation terminal;

[0014] The first receiving module is used to receive the target model parameters returned by the server, and update the initial network model according to the target model parameters to obtain the target network model for marking the region of the abnormal object.

[0015] Wherein, the first sending module includes:

[0016] An index determination unit, used to determine a regional statistical index associated with the original image based on the predicted annotation area and the standard annotation area;

[0017] The parameter adjustment unit is used to adjust the model parameters of the initial network model if the regional statistical indicators meet the model update conditions, obtain the first updated model parameters, and send the first updated model parameters to the server.

[0018] The number of original images is at least two, and the at least two original images include the original image T i ; i is a positive integer less than or equal to the number of images of at least two original images; original image T i The number of regions corresponding to the predicted annotation area is N, and the original image T i The number of corresponding standard marked areas is M; N is a positive integer, and M is a positive integer;

[0019] The indicator determination unit includes:

[0020] The first fusion subunit is used to fusion the original image T i The corresponding N predicted and labeled regions are regionally fused to obtain the predicted fused regions corresponding to the N predicted and labeled regions;

[0021] The second fusion subunit is used to fusion the original image T i The corresponding M standard annotated areas are regionally fused to obtain the standard fused areas corresponding to the M standard annotated areas;

[0022] The overlap determination subunit is used to determine the overlap between the predicted fusion area and the standard fusion area as the original image T i The corresponding first region overlap;

[0023] The index determination subunit is used to determine the regional statistical index associated with at least two original images according to the overlap degree of the first region corresponding to each original image.

[0024] Among them, the image corresponding to the predicted annotation area is the predicted area image, and the image corresponding to the standard annotation area is the standard area image;

[0025] The overlap determination subunit is specifically used to obtain the position information L of the predicted fusion area in the predicted area image. 1 , obtain the position information L of the standard fusion area in the standard area image 2 ;

[0026] The overlap determination subunit is also specifically used to determine the overlap according to the position information L 1 and location information L 2 , determine the overlap position information L between the predicted fusion area and the standard fusion area 3 ;

[0027] The overlap determination subunit is also specifically used to determine the overlap according to the position information L 1 , location information 2 and coincidence position information L 3 , determine the regional overlap between the predicted fusion region and the standard fusion region;

[0028] The overlap determination subunit is also specifically used to determine the region overlap as the original image T i The corresponding first area overlap.

[0029] The indicator determination subunit is specifically used to average the first region overlaps corresponding to each original image to obtain the average region overlaps corresponding to at least two original images;

[0030] The indicator determination subunit is further specifically configured to determine the average region overlap as a region statistical indicator associated with at least two original images.

[0031] The indicator determination subunit is specifically used to obtain the first area coincidence greater than the first coincidence threshold value from the first area coincidences corresponding to each original image, and count the number of first area coincidences greater than the first coincidence threshold value;

[0032] The index determination subunit is further specifically used to determine the index coincidence ratio corresponding to at least two original images based on the coincidence number and the number of images of at least two original images;

[0033] The indicator determination subunit is further specifically configured to determine the indicator coincidence ratio as a regional statistical indicator associated with at least two original images.

[0034] Wherein, the parameter adjustment unit includes:

[0035] A position acquisition subunit, used for acquiring standard region position information of the standard marked region and predicted region position information of the predicted marked region if the regional statistical index meets the model update condition;

[0036] A loss value determination subunit, used to determine the regional loss value of the initial network model according to the standard regional location information and the predicted regional location information;

[0037] A parameter adjustment subunit is used to adjust the model parameters in the initial network model according to the regional loss value, and when the adjusted initial network model meets the model convergence condition, the adjusted initial network model is determined as the updated network model;

[0038] The parameter sending subunit is used to obtain a first updated model parameter for updating the network model, and send the first updated model parameter to the server.

[0039] Among them, the model acquisition module includes:

[0040] A feature extraction unit, used for the first annotation terminal to obtain an initial network model from a server, input the original image into the initial network model, and extract an image feature vector of the original image through the initial network model;

[0041] A position determination unit, used to determine the predicted area position information associated with the abnormal object in the original image according to the image feature vector;

[0042] The region labeling unit is used to label the region of the original image according to the predicted region position information, and generate a predicted labeling region for labeling abnormal objects.

[0043] Wherein, the number of the annotated objects is at least two;

[0044] The regional audit modules include:

[0045] An image acquisition unit, used to acquire at least two candidate region images provided by at least two annotated objects; each of the at least two candidate region images includes a candidate annotated region for annotating an abnormal object in an original image;

[0046] A region review unit, used to determine a region review result according to the candidate annotated regions included in each candidate region image and the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image;

[0047] The area determination unit is used to determine a standard marking area for marking abnormal objects according to the area review result.

[0048] Among them, the regional audit units include:

[0049] A quantity comparison subunit is used to determine the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image, and to compare the numbers of at least two candidate regions to obtain a quantity comparison result;

[0050] A first determination subunit is configured to determine the at least S identical candidate region numbers as the target candidate region number if the quantity comparison result indicates that there are at least S identical candidate region numbers in the at least two candidate region numbers, and determine the region audit result according to the candidate region images associated with the target candidate region number; S is a positive integer less than or equal to the number of images of the at least two candidate region images;

[0051] The second determination subunit is used to determine the candidate annotated areas in at least two candidate area images as area review results if the quantity comparison result indicates that there are not at least S identical candidate area quantities in at least two candidate area quantities.

[0052] The first determination subunit is specifically used to obtain the candidate annotation areas included in every two candidate area images in the target candidate area image, and determine the second area overlap between the candidate annotation areas included in every two candidate area images; the target candidate area image is a candidate area image associated with the number of target candidate areas;

[0053] The first determination subunit is further specifically used to compare the second region coincidence with the second coincidence threshold value to obtain a coincidence comparison result;

[0054] The first determination subunit is further specifically used for merging the candidate annotation areas included in at least P candidate area images to obtain a merged candidate area, and determining the merged candidate area as the area review result if the overlap comparison result indicates that in the target candidate area image, there are at least P candidate area images whose second area overlap corresponding to every two candidate area images is greater than a second overlap threshold; P is a positive integer less than or equal to the number of images of the target candidate area image;

[0055] The first determination subunit is also specifically used to determine the candidate annotation area included in the target candidate area image as the area review result if the overlap comparison result indicates that in the target candidate area image, there is no second area overlap corresponding to every two candidate area images in at least P candidate area images greater than a second overlap threshold.

[0056] The area determination unit includes:

[0057] A first audit subunit is configured to send the regional audit result to the first audit terminal if the regional audit result is a candidate annotation region in at least two candidate region images, so that the first audit terminal determines a first candidate audit region for sending to the second audit terminal based on the candidate annotation regions in at least two candidate region images; the first candidate audit region is used to instruct the second audit terminal to return a standard annotation region for annotating abnormal objects;

[0058] The second audit subunit is used to send the regional audit result to the first audit terminal if the regional audit result is a candidate annotation area included in the target candidate area image, so that the first audit terminal determines a second candidate audit area for sending to the second audit terminal based on the candidate annotation area included in the target candidate area image; the second candidate audit area is used to instruct the second audit terminal to return a standard annotation area for annotating abnormal objects;

[0059] The third audit subunit is used to send the regional audit result to the second audit terminal if the regional audit result is a merged candidate region, so that the second audit terminal returns a standard marking region for marking abnormal objects according to the merged candidate region.

[0060] An embodiment of the present application provides an image data processing method, including:

[0061] The server sends the initial network model to the first annotation terminal and the second annotation terminal, so that the first annotation terminal performs region annotation on the original image based on the initial network model to obtain a predicted annotation region for annotating abnormal objects in the original image;

[0062] Receive a first updated model parameter returned by the first annotation terminal and a second updated model parameter returned by the second annotation terminal; the first updated model parameter is obtained by adjusting the model parameter of the initial network model by the first annotation terminal based on the predicted annotation area and the standard annotation area for annotating the abnormal object; the standard annotation area is obtained by performing an area review on the candidate annotation area after obtaining the candidate annotation area provided by the annotation object for annotating the abnormal object in the original image;

[0063] The first update model parameters and the second update model parameters are fused to generate target model parameters, and the target model parameters are sent to the first labeling terminal and the second labeling terminal; the target model parameters are used to instruct the first labeling terminal to update the initial network model through the target model parameters to obtain a target network model for regional labeling of abnormal objects.

[0064] An embodiment of the present application provides an image data processing device, including:

[0065] A model sending module, used for the server to send the initial network model to the first annotation terminal and the second annotation terminal, so that the first annotation terminal performs region annotation on the original image based on the initial network model to obtain a predicted annotation region for annotating abnormal objects in the original image;

[0066] The second receiving module is used to receive the first updated model parameter returned by the first annotation terminal and the second updated model parameter returned by the second annotation terminal; the first updated model parameter is obtained by adjusting the model parameter of the initial network model by the first annotation terminal based on the predicted annotation area and the standard annotation area for annotating the abnormal object; the standard annotation area is obtained by performing regional review on the candidate annotation area after obtaining the candidate annotation area provided by the annotation object for annotating the abnormal object in the original image;

[0067] The second sending module is used to perform parameter fusion on the first update model parameters and the second update model parameters to generate target model parameters, and send the target model parameters to the first labeling terminal and the second labeling terminal; the target model parameters are used to instruct the first labeling terminal to update the initial network model through the target model parameters to obtain a target network model for regional labeling of abnormal objects.

[0068] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0069] The processor is connected to the memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the computer device executes the method provided in the embodiment of the present application.

[0070] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0071] In one aspect, the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the present application.

[0072] In an embodiment of the present application, when the first annotation terminal obtains the initial network model from the server, it can perform regional annotation on the original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the original image. Further, the first annotation terminal can obtain a candidate annotation area provided by the annotation object for annotating abnormal objects in the original image, perform regional review on the candidate annotation area, and obtain a standard annotation area for annotating abnormal objects. Further, the first annotation terminal can adjust the model parameters of the initial network model based on the predicted annotation area and the standard annotation area to obtain a first updated model parameter, and send the first updated model parameter to the server so that the server performs parameter fusion on the first updated model parameter and the second updated model parameter to generate a target model parameter. Among them, the second updated model parameter is provided by the second annotation terminal. Further, the first annotation terminal can receive the target model parameter returned by the server, update the initial network model through the target model parameter, and obtain a target network model for regional annotation of abnormal objects. It can be seen that the embodiment of the present application proposes an auxiliary annotation method based on federated learning, which indicates that the first annotation terminal can train the initial network model in the first annotation terminal based on the predicted annotation area and the standard annotation area corresponding to the original image, and then send the first updated model parameters obtained by the training to the server. Among them, the predicted annotation area here can be the area output by the network model, and the standard annotation area here can be the area determined by the annotation object. Therefore, the embodiment of the present application can use the new annotation data of multiple annotators (i.e., the first annotation object corresponding to the first annotation terminal and the second annotation object corresponding to the second annotation terminal) to timely train the initial network model on the basis of protecting the data privacy of the data to be annotated (i.e., the original image), and improve the ability of the model to adapt to new data. In addition, the present application can obtain the target model parameters by aggregating the updated model parameters obtained by training multiple annotators (i.e., the first updated model parameters obtained by training the first annotation terminal and the second updated model parameters obtained by training the second annotation terminal), and the first annotation terminal and the second annotation terminal can improve the accuracy of the auxiliary annotation of the target network model when updating the initial network model through the target model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0075] Figure 2 This is a schematic diagram of a scenario for data interaction provided by an embodiment of the present application;

[0076] Figure 3 It is a flowchart of an image data processing method provided in an embodiment of the present application;

[0077] Figure 4a is a schematic diagram of a scenario for determining the regional overlap of a single region provided in an embodiment of the present application;

[0078] Figure 4b is a schematic diagram of a scenario for determining the regional overlap of multiple regions provided in an embodiment of the present application;

[0079] Figure 5 It is a schematic diagram of a multi-annotator annotation process provided in an embodiment of the present application;

[0080] Figure 6 It is a flowchart of an image data processing method provided in an embodiment of the present application;

[0081] Figure 7 is a schematic diagram of a scenario for determining a merging candidate region provided in an embodiment of the present application;

[0082] Figure 8a It is a schematic diagram of a process for multiple doctors to determine a standard area image provided by an embodiment of the present application;

[0083] Figure 8b It is a schematic diagram of a process for a single doctor to determine a standard area image provided by an embodiment of the present application;

[0084] Fig. 9 It is a flowchart of an image data processing method provided in an embodiment of the present application;

[0085] Fig.10 It is a schematic diagram of a flow chart of model training provided in an embodiment of the present application;

[0086] Fig.11 is a structural schematic diagram of an image data processing device provided in an embodiment of the present application;

[0087] Fig.12 is a structural schematic diagram of an image data processing device provided in an embodiment of the present application;

[0088] Fig.13 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0089] Fig.14 It is a structural diagram of an image data processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0091] It should be understood that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0092] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0093] Among them, the solution provided in the embodiments of the present application mainly involves computer vision (CV) technology and machine learning (ML) technology of artificial intelligence.

[0094] Among them, computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0095] Among them, machine learning is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. Among them, deep learning technology is a technology that uses deep neural network systems to perform machine learning.

[0096] It should be understood that federated machine learning (i.e., federated learning) is a machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. Federated learning is essentially a distributed machine learning technology, or a machine learning framework.

[0097] For details, see Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 2000 and a user terminal cluster. The user terminal cluster may specifically include one or more user terminals, and the number of user terminals in the user terminal cluster is not limited. Figure 1As shown, the multiple user terminals may specifically include user terminal 3000a, user terminal 3000b, user terminal 3000c, ..., user terminal 3000n; user terminal 3000a, user terminal 3000b, user terminal 3000c, ..., user terminal 3000n may respectively be directly or indirectly connected to the business server 2000 via wired or wireless communication, so that each user terminal may exchange data with the business server 2000 via the network connection.

[0098] Among them, the business server 2000 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0099] Each user terminal in the user terminal cluster may include: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home, a wearable device, a vehicle terminal, and other smart terminals with image data processing functions. Figure 1 Select one user terminal from the multiple user terminals shown as the first marking terminal. Figure 1 The user terminal 3000n shown is used as the first annotation terminal. For ease of understanding, the embodiment of the present application can be Figure 1 Select one user terminal from the multiple user terminals shown as the second annotation terminal. Figure 1 The user terminal 3000c shown serves as the second annotation terminal.

[0100] It is understandable that the embodiment of the present application can determine the annotation object corresponding to the first annotation terminal as the first annotation object, and the first annotation object is the user using the first annotation terminal. It should be understood that the first annotation terminal can obtain the original image, and the original image associated with the first annotation terminal can be the first original image. The first annotation object in the embodiment of the present application can annotate abnormal objects in the first original image through the first annotation terminal.

[0101] It is understandable that the embodiment of the present application can determine the annotation object corresponding to the second annotation terminal as the second annotation object, and the second annotation object is the user using the second annotation terminal. It should be understood that the second annotation terminal can obtain the original image, and the original image associated with the second annotation terminal can be the second original image. The second annotation object in the embodiment of the present application can annotate the abnormal object in the second original image through the second annotation terminal.

[0102] Optionally, the embodiment of the present application may also be Figure 1 A user terminal is selected from the multiple user terminals shown as the third annotation terminal, and the annotation object corresponding to the third annotation terminal can be a third annotation object. For example, in the embodiment of the present application, Figure 1 The user terminal 3000a shown is used as the third annotation terminal. It can be understood that the first annotation terminal, the second annotation terminal and the third annotation terminal can be collectively referred to as annotation terminals, and the embodiment of the present application does not limit the number of annotation terminals.

[0103] It should be understood that when the annotation terminal includes a first annotation terminal and a second annotation terminal, the business server 2000 can send the initial network model to the first annotation terminal and the second annotation terminal. In this way, the first annotation terminal can perform regional annotation on the first original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the first original image; the second annotation terminal can perform regional annotation on the second original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the second original image. Further, the first annotation terminal can obtain a candidate annotation area for annotating abnormal objects in the first original image provided by the first annotation object, perform regional review on the candidate annotation area corresponding to the first original image, and obtain a standard annotation area for annotating abnormal objects in the first original image; the second annotation terminal can obtain a candidate annotation area for annotating abnormal objects in the second original image provided by the second annotation object, perform regional review on the candidate annotation area corresponding to the second original image, and obtain a standard annotation area for annotating abnormal objects in the second original image. Furthermore, the first annotation terminal can adjust the model parameters of the initial network model based on the candidate annotation area and the standard annotation area corresponding to the first original image to obtain the first updated model parameters, and then send the first updated model parameters to the business server 2000; the second annotation terminal can adjust the model parameters of the initial network model based on the candidate annotation area and the standard annotation area corresponding to the second original image to obtain the second updated model parameters, and then send the second updated model parameters to the business server 2000. In this way, the business server can perform parameter fusion on the first updated model parameters returned by the first annotation terminal and the second updated model parameters returned by the second annotation terminal to generate target model parameters, and then send the target model parameters to the first annotation terminal and the second annotation terminal, so that the first annotation terminal and the second annotation terminal can respectively update the initial network model through the target model parameters to obtain the target network model for regional annotation of abnormal objects.

[0104] It can be understood that the initial network model involved in the model update of the first annotation terminal (i.e., the first initial network model) and the initial network model involved in the model update of the second annotation terminal (i.e., the second initial network model) can be the same or different. When the first initial network model and the second initial network model are the same, the first initial network model and the second initial network model can be obtained by the first annotation terminal and the second annotation terminal from the business server 2000 at the same time; the first initial network model and the second initial network model can also be obtained by the first annotation terminal and the second annotation terminal updating their respective local network models at the previous moment before the current moment. When the first initial network model and the second initial network model are different, one of the first initial network model and the second initial network model can be obtained by the first annotation terminal or the second annotation terminal from the business server 2000, and the other initial network model can be obtained by the second annotation terminal or the first annotation terminal updating the local network model at the previous moment before the current moment; the first initial network model and the second initial network model can also be obtained by the first annotation terminal and the second annotation terminal updating their respective local network models at different moments before the current moment.

[0105] Among them, the predicted annotation area, candidate annotation area and standard annotation area involved in the embodiments of the present application can also be referred to as a candidate box, which is a professional term often used in target detection and is used to frame the position of the candidate target area.

[0106] Among them, the abnormal object in the embodiment of the present application can be a part of the original image that the annotation object is interested in, for example, a diseased part in a medical image.

[0107] For further understanding, please refer to Figure 2 , Figure 2 Schematic diagram of a scenario for data interaction provided by an embodiment of the present application. Figure 2 The server 20a shown may be the above Figure 1 The service server 2000 in the corresponding embodiment is as follows: Figure 2 The user terminal 20b shown may be the above Figure 1 The first marking terminal of the corresponding embodiment is as follows Figure 2 The user terminal 20c shown may be the above Figure 1 The second annotation terminal of the corresponding embodiment. Among them, the user corresponding to the user terminal 20b can be the first annotation object, and the user corresponding to the user terminal 20c can be the second annotation object.

[0108] like Figure 2As shown, the server 20a can send the initial network model to the user terminal 20b and the user terminal 20c, so that the user terminal 20b can obtain the original image associated with the user terminal 20b (i.e., the first original image), where the number of original images can be n, where n can be a positive integer. When n is a positive integer greater than 1, the n original images can include the original image T 1 , ..., original image T n , the original image T 1 , ..., original image T n can be collectively referred to as the first original image. Further, the user terminal 20b can 1 , ..., original image T n are respectively input into the initial network model, and the original image T is 1 , ..., original image T n Perform region annotation to obtain the original image T 1 The corresponding prediction area image U 1 , ..., original image T n The corresponding prediction area image U n Among them, the predicted area image U 1 Including for annotating the original image T 1 The predicted annotation area N of the abnormal object in 1 (not shown in the figure), ..., prediction area image U n Including for annotating the original image T n The predicted annotation area N of the abnormal object in n (Not shown in the figure).

[0109] like Figure 2 As shown, the user terminal 20b can obtain the first annotated object for the original image T 1 The number of the first annotation objects provided in the candidate region image 21a may be m, where m may be a positive integer. The candidate region image 21a may include m candidate region images. When m is a positive integer greater than 1, the candidate region images provided by the m first annotation objects may include candidate region image 22a, candidate region image 22b, ..., candidate region image 22m. The candidate region image 22a includes the image used to annotate the original image T 1 The candidate annotated region 24a (not shown in the figure) of the abnormal object in the original image T is included in the candidate region image 22b. 1 The candidate labeled regions 24b (not shown in the figure) of the abnormal objects in the original image T are included in the candidate labeled regions 24b, ..., and the candidate region images 22m. 1 The candidate labeled area 24m (not shown in the figure) of the abnormal object in .

[0110] Similarly, the user terminal 20b can obtain the first annotated object for the original image T n The candidate region image 21n provided may include m candidate region images, and the candidate region images provided by the m first annotation objects may include candidate region image 23a, candidate region image 23b, ..., candidate region image 23m. Among them, the candidate region image 23a includes the image used to annotate the original image T n The candidate annotated region 25a (not shown in the figure) of the abnormal object in the original image T is included in the candidate region image 23b. n The candidate annotated regions 25b (not shown in the figure) of the abnormal objects in the original image T are included. n The candidate labeled area 25m (not shown in the figure) of the abnormal object in the image.

[0111] The first annotation object can be based on the original image T 1 , for the prediction area image U 1 The predicted annotation area N in 1 The candidate region image 21a is obtained by correction; similarly, the first annotated object can be obtained according to the original image T n , for the prediction area image U n The predicted annotation area N in n The candidate region image 21n is obtained by correction. Optionally, the first annotation object can also be directly 1 The region is annotated to obtain the candidate region image 21a; similarly, the first annotated object can also be directly annotated on the original image T n Perform region labeling to obtain a candidate region image 21n.

[0112] like Figure 2 As shown, the user terminal 20b can perform a region audit on the candidate region image 21a, that is, based on the candidate region images 22a, 22b, ..., 22m in the candidate region image 21a, the original image T 1 The corresponding standard area image F 1 Similarly, the user terminal 20b can perform a regional audit on the candidate region image 21n, that is, based on the candidate region images 23a, 23b, ..., 23m in the candidate region image 21n, obtain the standard region image n corresponding to the original image n. Among them, the standard region image F 1 Including for annotating the original image T 1 The standard annotation area M of the abnormal object in 1 (not shown in the figure), ..., standard area image Fn Including for annotating the original image T n The standard annotation area M of the abnormal object in n (Not shown in the figure).

[0113] like Figure 2 As shown, the user terminal 20b can adjust the model parameters of the initial network model in the user terminal 20b (i.e., perform model training on the initial network model) based on the predicted area image (or the predicted annotated area in the predicted area image) corresponding to the first original image and the standard area image (or the standard annotated area in the standard area image) corresponding to the first original image, and obtain an updated network model (i.e., a first updated network model). The model parameters of the first updated network model can be the first updated model parameters. Further, the user terminal 20b can send the first updated model parameters to the server 20a so that the server 20a returns the updated model parameters.

[0114] Similarly, the user terminal 20c can obtain the original image (i.e., the second original image) associated with the user terminal 20c, and perform region annotation on the second original image based on the initial network model to obtain the predicted region image corresponding to the second original image. The predicted region image corresponding to the second original image includes a predicted annotation region for annotating abnormal objects in the second original image. Further, the user terminal 20c can obtain the candidate region image provided by the second annotation object, and then perform a region audit on the candidate region image corresponding to the second original image to obtain the standard region image corresponding to the first original image. Further, the user terminal 20c can adjust the model parameters of the initial network model in the user terminal 20c based on the predicted region image corresponding to the second original image and the standard region image corresponding to the second original image to obtain an updated network model (i.e., a second updated network model). The model parameters of the second updated network model can be second updated model parameters. Further, the user terminal 20c can send the second updated model parameters to the server 20a so that the server 20a returns the updated model parameters.

[0115] Furthermore, if Figure 2As shown, when the server 20a receives the first updated model parameter sent by the user terminal 20b and the second updated model parameter sent by the user terminal 20c, it can fuse the first updated model parameter and the second updated model parameter to obtain the target model parameter, and then return the target model parameter to the user terminal 20b and the user terminal 20c. In this way, after receiving the target model parameter, the user terminal 20b can use the target model parameter as the updated model parameter to update the initial network model in the user terminal 20b with the updated model parameter to obtain the target network model; after receiving the target model parameter, the user terminal 20c can use the target model parameter as the updated model parameter to update the initial network model in the user terminal 20c with the updated model parameter to obtain the target network model.

[0116] The specific implementation method of data interaction between the server 20a and the user terminal 20b can be as follows: Figure 3-Figure 10 Description of data interaction between the server and the first annotation terminal in the corresponding embodiment.

[0117] It can be seen that the annotation terminal in the embodiment of the present application (for example, the first annotation terminal) can obtain the initial network model from the server, and then train the initial network model based on the predicted annotation area obtained by the initial network model and the standard annotation area determined by the annotation object to obtain updated model parameters (for example, first updated model parameters). In this way, the server can aggregate the updated model parameters provided by different annotation terminals (for example, the first updated model parameters provided by the first annotation terminal and the second updated model parameters provided by the second annotation terminal) to obtain the target model parameters. Therefore, when the first annotation terminal updates the initial network model through the target model parameters, the accuracy of the auxiliary annotation of the network model (i.e., the target network model) can be improved.

[0118] For further information, see Figure 3 , Figure 3 is a flowchart of an image data processing method provided by an embodiment of the present application. The method can be executed by a server, or by a first annotation terminal, or by both the server and the first annotation terminal. The server can be the above-mentioned Figure 2 In the corresponding implementation of the server 20a, the first annotation terminal may be the above Figure 2 The corresponding user terminal 20b in the embodiment. For ease of understanding, this embodiment takes the method executed by the first annotation terminal as an example for explanation. The image data processing method may include the following steps S101-S104:

[0119] Step S101, obtaining an initial network model from a server, and annotating regions of an original image based on the initial network model to obtain predicted annotated regions for annotating abnormal objects in the original image;

[0120] Specifically, the first annotation terminal can obtain an initial network model from a server, input the original image (i.e., the first original image) into the initial network model, and extract the image feature vector of the original image through the initial network model. Furthermore, the first annotation terminal can determine the predicted region position information associated with the abnormal object in the original image based on the image feature vector. Furthermore, the first annotation terminal can perform region annotation on the original image based on the predicted region position information to generate a predicted annotation region for annotating the abnormal object.

[0121] Step S102, obtaining a candidate annotation region provided by the annotation object for annotating an abnormal object in the original image, performing a region review on the candidate annotation region, and obtaining a standard annotation region for annotating the abnormal object;

[0122] It can be understood that the number of the original images may be at least two, and the at least two original images include the original image T i , where i can be a positive integer less than or equal to the number of images of at least two original images. i The number of regions corresponding to the predicted annotation region can be N, and the original image T i The number of corresponding standard marked areas may be M, where N may be a positive integer and M may be a positive integer.

[0123] Optionally, the number of original images may be one, and one original image may be original image T 1 Among them, the original image T 1 The number of regions corresponding to the predicted annotation region can be N, and the original image T 1 The number of corresponding standard annotated areas may be M. For ease of understanding, the present embodiment of the application is described by taking the example that the number of original images is at least two.

[0124] It should be understood that N and M in the embodiment of the present application may be the same or different, that is, for each of the at least two original images, the number of predicted annotation areas and standard annotation areas corresponding to each original image may be the same or different. The embodiment of the present application does not limit the specific values ​​of N and M.

[0125] Step S103, adjusting the model parameters of the initial network model based on the predicted annotated area and the standard annotated area to obtain first updated model parameters, and sending the first updated model parameters to the server;

[0126] Specifically, the first annotation terminal can determine the regional statistical index associated with the original image based on the predicted annotation area and the standard annotation area. Further, if the regional statistical index meets the model update condition, the first annotation terminal can adjust the model parameters of the initial network model to obtain the first updated model parameters, and send the first updated model parameters to the server.

[0127] It can be understood that the image associated with the first annotation terminal can be an annotated image, and the annotated image can include the original image associated with the first annotation terminal. The annotated image can be understood as all images that have been annotated by the first annotation terminal in history, and the original image can be understood as an image that meets the image acquisition condition in the annotated image. The image acquisition condition can indicate that the first annotation terminal acquires the most recently annotated H 1 images (for example, 100), where H 1 It can be a positive integer, that is, the first annotation terminal can be the most recently marked H 1 Optionally, the image acquisition condition may also indicate that the first annotation terminal acquires H within a time period. 2 images, where H 2 It can be a positive integer, that is, the first marking terminal can 2 An image is taken as the original image. The time period may be in hours, minutes, or seconds, and the unit of the time period is not limited here. It should be understood that the embodiment of the present application does not limit the original image obtained by the image acquisition condition.

[0128] Optionally, when the regional statistical indicators do not meet the model update conditions, the first annotation terminal can also adjust the model parameters of the initial network model, that is, when the annotation object wants to update the model, the first annotation terminal can adjust the model parameters of the initial network model.

[0129] It should be understood that the specific process of the first annotation terminal determining the regional statistical index associated with the original image based on the predicted annotation area and the standard annotation area can be described as follows: the first annotation terminal can i The corresponding N predicted annotation regions are subjected to regional fusion to obtain the predicted fusion region corresponding to the N predicted annotation regions. i The corresponding M standard annotated regions are regionally fused to obtain the standard fused region corresponding to the M standard annotated regions. Further, the first annotation terminal can determine the region overlap between the predicted fused region and the standard fused region as the original image T iCorresponding first region coincidence. Further, the first annotation terminal may determine a region statistical index associated with at least two original images according to the first region coincidence corresponding to each original image.

[0130] The image corresponding to the predicted annotation area may be the predicted area image, and the image corresponding to the standard annotation area may be the standard area image. It can be understood that the regional fusion can convert the original image T i All regions in the corresponding prediction region image (i.e., N prediction annotation regions) are regarded as a whole (i.e., prediction fusion region). Regional fusion can transform the original image T i All regions in the corresponding standard region image (i.e., M standard annotated regions) are regarded as a whole (i.e., standard fusion region). Therefore, the first annotation terminal can determine the region overlap between the predicted fusion region and the standard fusion region, and then determine the region overlap between the predicted fusion region and the standard fusion region as the original image T i The corresponding first area overlap.

[0131] It should be understood that the specific process of the first annotation terminal determining the region overlap between the predicted fusion region and the standard fusion region can be described as follows: the first annotation terminal can obtain the position information L of the predicted fusion region in the predicted region image. 1 , obtain the position information L of the standard fusion area in the standard area image 2 Furthermore, the first marking terminal can 1 and location information L 2 , determine the overlap position information L between the predicted fusion area and the standard fusion area 3 Furthermore, the first marking terminal can 1 , location information 2 and coincidence position information L 3 , determine the regional overlap between the predicted fusion area and the standard fusion area. 1 , location information 2 and coincidence position information L 3 , the union position information L between the predicted fusion area and the standard fusion area can be determined 4 .

[0132] It can be understood that the first annotation terminal can determine the area overlap between the predicted fusion area and the standard fusion area through the intersection-over-Union (IoU), and the effect of auxiliary annotation can be quantified through the intersection-over-Union. IoU is a concept used in target detection, which is the overlap rate of the generated candidate box and the original marked box, that is, the ratio of their intersection to their union. Among them, the candidate box can be expressed as the standard fusion area, the original marked box can be expressed as the predicted fusion area, and the overlap rate of the standard fusion area and the predicted fusion area can be referred to in formula (1):

[0133]

[0134] Among them, ROI_1 can represent the standard fusion region, and ROI_2 can represent the predicted fusion region. Therefore, ROI_1∩ROI_2 can represent the intersection of the standard fusion region and the predicted fusion region, and ROI_1∪ROI_2 can represent the union of the standard fusion region and the predicted fusion region.

[0135] For easier understanding, see Figure 4a , Figure 4a is a schematic diagram of a scenario for determining the regional overlap of a single region provided in an embodiment of the present application. Figure 4a The image 40a shown may be a prediction area image, such as Figure 4a The image 40b shown may be a standard region image. The predicted region image 40a may include N predicted annotated regions. Here, the predicted region image 40a includes one predicted annotated region as an example. The predicted annotated region may be Figure 4a The standard region image 40b may include M standard annotated regions. Here, the standard region image 40b includes one standard annotated region as an example for explanation. The one standard annotated region may be Figure 4a Therefore, the first annotation terminal can directly use region 41a as the predicted fusion region and region 41b as the standard fusion region, and then determine the region overlap between region 41a and region 41b as the first region overlap corresponding to the original image corresponding to the predicted region image 40a and the standard region image 40b (for example, region overlap C 1 ).

[0136] Among them, it can be understood that, except for the difference in the marked areas respectively included in the predicted area image and the standard area image, there is no difference in other aspects (because the predicted area image and the standard area image are both generated by the original image), so the upper left corner of the predicted area image and the standard area image is used as the coordinate origin, the x-axis to the right of the origin, and the y-axis downward from the origin, and the coordinates generated respectively are consistent. Therefore, the position information corresponding to the abnormal objects in the predicted area image and the standard area image is corresponding.

[0137] According to the above coordinates, if Figure 4a The position information of the area 41a shown may be the position information L 11 ,like Figure 4a The position information of the area 41b shown may be the position information L 12 Therefore, the position information L 11 and location information L 12 The intersection position information (i.e., the overlap position information) of 13 , location information L 11 and location information L 12 The union position information can be L 14 .

[0138] It is understandable that the first annotation terminal can determine the region overlap between region 41a and region 41b according to the above formula (1). ROI_1 can represent region 41b, which is the region 41b obtained by position information L 12 ROI_2 can represent the region 41a, which is determined by the position information L 11 ROI_1∩ROI_2 can represent the intersection area of ​​area 41a and area 41b, which is determined by the intersection position information L13; ROI_1∪ROI_2 can represent the union area of ​​area 41a and area 41b, which is determined by the union position information L 14 Determined.

[0139] like Figure 4a As shown, region 42a and region 42c may be region 41a (i.e., ROI_2), region 42a and region 42b may be region 41b (i.e., ROI_1), region 42a may be the intersection region of region 41a and region 41b (i.e., ROI_1∩ROI_2), and region 42a, region 42b, and region 42c may be the union region of region 41a and region 41b (i.e., ROI_1∪ROI_2).

[0140] For easier understanding, see Figure 4b , Figure 4b is a schematic diagram of a scenario for determining the regional overlap of multiple regions provided in an embodiment of the present application. Figure 4b The image 40a shown may be Figure 4a The prediction area image 40a in the corresponding embodiment is as follows: Figure 4b The image 40c shown may be a standard region image. The standard region image 40c may include M standard annotated regions. Here, an example is taken in which the annotated region image 40c includes three standard annotated regions. The three standard annotated regions may be Figure 4b Therefore, the first annotation terminal can use region 41a as the predicted fusion region, and use regions 41b, 41c, and 41d as the standard fusion region, and then determine the region overlap between the predicted fusion region and the standard fusion region as the first region overlap corresponding to the original image corresponding to the predicted region image 40a and the standard region image 40c (for example, region overlap C 2 ).

[0141] According to the above coordinates, if Figure 4b The position information of the area 41a shown may be the position information L 11 ,like Figure 4b The position information of the area 41b, the area 41c and the area 41d shown may be the position information L 15 Therefore, the predicted fusion area (i.e., the position information L 11 ) and the standard fusion area (i.e., location information L 15 ) can be L 16 , the union position information of the predicted fusion area and the standard fusion area can be L 17 .

[0142] It is understandable that the first annotation terminal can determine the region overlap between the predicted fusion region and the standard fusion region according to the above formula (1). Wherein, ROI_1 can represent region 41b, region 41c and region 41d, where region 41b, region 41c and region 41d are determined by the position information L 15 ROI_2 can represent the region 41a, which is determined by the position information L 11 Determined; ROI_1∩ROI_2 can represent the intersection area of ​​the predicted fusion area and the standard fusion area, which is determined by the intersection position information L 16 Determined; ROI_1∪ROI_2 can represent the union area of ​​the predicted fusion area and the standard fusion area, which is determined by the union position information L 17 Determined.

[0143] like Figure 4bAs shown, region 42a and region 42c can be predicted fusion regions (i.e., ROI_2), region 42a, region 42b, region 42d, and region 42e can be standard fusion regions (i.e., ROI_1), region 42a can be the intersection region of the predicted fusion region and the standard fusion region (i.e., ROI_1∩ROI_2), and region 42a, region 42b, region 42c, region 42d, and region 42e can be the union region of the predicted fusion region and the standard fusion region (i.e., ROI_1∪ROI_2).

[0144] The at least two original images may also include the original image T j , where j can be a positive integer less than or equal to the number of images of at least two original images, and where j is not equal to i. It should be understood that the first annotation terminal determines the original image T j The specific process of the corresponding first region overlap can refer to the above determination of the original image T i The description of the corresponding first region overlap will not be repeated here.

[0145] It should be understood that the specific process of the first annotation terminal determining the regional statistical index associated with at least two original images can be described as follows: the first annotation terminal can average the first regional coincidence corresponding to each original image to obtain the average regional coincidence corresponding to the at least two original images. Further, the first annotation terminal can determine the average regional coincidence as the regional statistical index associated with the at least two original images.

[0146] Optionally, it should be understood that the specific process of the first annotation terminal determining the regional statistical index associated with at least two original images can also be described as: the first annotation terminal can obtain the first region coincidence greater than the first coincidence threshold value in the first region coincidence corresponding to each original image, and count the number of coincidences of the first region coincidence greater than the first coincidence threshold value. Among them, the embodiment of the present application does not limit the specific value of the first coincidence threshold value. Further, the first annotation terminal can determine the indicator coincidence ratio corresponding to at least two original images based on the number of coincidences and the number of images of at least two original images. Further, the first annotation terminal can determine the indicator coincidence ratio as the regional statistical index associated with at least two original images.

[0147] Among them, it can be understood that the first annotation terminal can use the ratio of the number of overlaps and the number of images of at least two original images as the indicator overlap ratio, where the indicator overlap ratio can be a decimal, the indicator overlap ratio can be a percentage, or the indicator overlap ratio can be a fraction. The expression form of the indicator overlap ratio is not limited here.

[0148] For easier understanding, see Figure 5 , Figure 5 1 is a schematic diagram of a multi-annotator annotation process provided by an embodiment of the present application. Figure 5 The data shown is T 1 、Data T 2 ,…, data T N The image data (i.e., original images) of different annotators can be read by the first annotation terminal into the annotation system, and the image data can be predicted (i.e., auxiliary annotation) by using the artificial intelligence model (i.e., the initial network model), and their respective prediction results (i.e., auxiliary annotation results, predicted area images) can be generated. 1 It can be data T 1 Corresponding predicted area image, auxiliary annotation result U 2 It can be data T 2 The corresponding predicted area image, ..., auxiliary annotation result U N It can be data T N The corresponding prediction area image.

[0149] Among them, the annotation system in the embodiment of the present application can be used for medical data annotation systems for hospitals, scientific research institutes, enterprises and institutions, and the medical data annotation system can help researchers to quickly and efficiently annotate medical data. The annotation system can display the medical data and auxiliary annotation results of different annotators, and provide editing options for doctors (i.e., the annotated objects).

[0150] It is understandable that the purpose of AI-assisted labeling is to provide the prediction results of the AI ​​model as pre-labeling when doctors label data. Doctors can add, delete, modify and confirm the pre-labeling results to form the final labeling results (i.e., the labeling gold standard). 1 Can be used to assist in the annotation result U 1 The corresponding standard region image, annotated with gold standard F 2 Can be used to assist in the annotation result U 2 The corresponding standard region image, ..., annotated gold standard F N Can be used to assist in the annotation result U N The corresponding standard area image.

[0151] like Figure 5As shown in FIG. 1 , the first annotation terminal can compare and count the auxiliary annotation effects, that is, the auxiliary annotation results and the annotation gold standard between different annotators are counted according to specific rules (for example, the error between the auxiliary annotation results and the annotation gold standard), and whether the auxiliary annotation meets the standard is determined based on the statistical results and predefined rules. Here, the segmentation annotation task is used as an example to illustrate the statistical indicators (i.e., regional statistical indicators). For example, the first annotation terminal can compare the most recently annotated H 1 H in a (e.g., 100) patient or time period 2 The error between the auxiliary annotation results and the annotation gold standard is calculated for each patient (for example, 100 patients). Here, H 1 Can be a positive integer, where H 2 It can be a positive integer, and the error here can be obtained by IOU. Therefore, the first annotation terminal can 1 or H 2 The average IOU of the auxiliary annotation results of each patient and the annotation gold standard is used as a statistical indicator.

[0152] like Figure 5 As shown, if the auxiliary annotation result does not meet the standard, the first annotation terminal can start the automatic update of the artificial intelligence model, that is, based on the annotation gold standard of the participating annotation doctors and the corresponding medical data, the artificial intelligence model starts training / tuning based on federated learning to update the artificial intelligence model. After the training / tuning is completed, the new model will be used for auxiliary annotation. Among them, the training / tuning of the artificial intelligence model is run in the background and does not affect the current auxiliary annotation work. Optionally, if the auxiliary annotation result meets the standard, the first annotation terminal can use the original artificial intelligence model for auxiliary annotation.

[0153] It should be understood that the first annotation terminal can obtain a statistical indicator threshold associated with the regional statistical indicator. If the regional statistical indicator (i.e., the average regional overlap or the indicator overlap ratio) is less than the statistical indicator threshold, it is determined that the regional statistical indicator meets the model update condition. Optionally, if the regional statistical indicator is greater than or equal to the statistical indicator threshold, it is determined that the regional statistical indicator does not meet the model update condition.

[0154] It should be understood that the specific process of the first annotation terminal adjusting the parameters of the initial network model to obtain the first updated model parameters can be described as follows: if the regional statistical indicators meet the model update conditions, the first annotation terminal can obtain the standard regional location information of the standard annotation area and the predicted regional location information of the predicted annotation area. Further, the first annotation terminal can determine the regional loss value of the initial network model based on the standard regional location information and the predicted regional location information. Further, the first annotation terminal can adjust the model parameters in the initial network model based on the regional loss value, and when the adjusted initial network model meets the model convergence conditions, the adjusted initial network model is determined as the updated network model. Further, the first annotation terminal can obtain the first updated model parameters of the updated network model and send the first updated model parameters to the server.

[0155] Optionally, if the regional statistical index does not meet the model update condition, the first labeling terminal may not need to iteratively train the initial network model, but continue to label the region of the abnormal object based on the initial network model.

[0156] Step S104, receiving the target model parameters returned by the server, and updating the initial network model by the target model parameters to obtain a target network model for marking the region of the abnormal object.

[0157] It can be understood that after obtaining the target network model, the first annotation terminal can obtain a new original image associated with the first annotation terminal, perform region annotation on the new original image based on the target network model, and obtain a predicted annotation area for annotating abnormal objects in the new original image.

[0158] It should be understood that the initial network model and the target network model can be collectively referred to as the generalized network model, and the initial network model and the target network model are the names of the generalized network model at different times. In the training phase, the generalized network model can be called the initial network model, and in the prediction phase, the generalized network model can be called the target network model.

[0159] It can be seen that the embodiment of the present application proposes an auxiliary annotation method based on federated learning, which indicates that the first annotation terminal can train the initial network model in the first annotation terminal based on the predicted annotation area and the standard annotation area corresponding to the original image, and then send the first updated model parameters obtained by the training to the server. Among them, the predicted annotation area here can be the area output by the network model, and the standard annotation area here can be the area determined by the annotation object. Therefore, the embodiment of the present application can use the new annotation data of multiple annotators (i.e., the first annotation object corresponding to the first annotation terminal and the second annotation object corresponding to the second annotation terminal) to timely train the initial network model on the basis of protecting the data privacy of the data to be annotated (i.e., the original image), and improve the ability of the model to adapt to new data. In addition, the present application can obtain the target model parameters by aggregating the updated model parameters obtained by training multiple annotators (i.e., the first updated model parameters obtained by training the first annotation terminal and the second updated model parameters obtained by training the second annotation terminal), and the first annotation terminal and the second annotation terminal can improve the accuracy of the auxiliary annotation of the target network model when updating the initial network model through the target model parameters.

[0160] For further information, see Figure 6 , Figure 6 is a flow chart of an image data processing method provided in an embodiment of the present application. The image data processing method may include the following steps S1021-S1023, and steps S1021-S1023 are Figure 3 A specific embodiment of step S102 in the corresponding embodiment.

[0161] Step S1021, obtaining at least two candidate region images provided by at least two annotated objects;

[0162] It is understandable that the annotation object can correct the predicted annotation area for annotating the abnormal object in the original image according to the original image to obtain the candidate annotation area for annotating the abnormal object in the original image. Optionally, the annotation object can also directly annotate the area of ​​the original image to obtain the candidate annotation area for annotating the abnormal object in the original image.

[0163] It can be understood that the number of annotated objects can be at least two, and each of the at least two annotated objects or part of the annotated objects can provide a candidate region image corresponding to the original image (e.g., the original image T) to obtain at least two candidate region images. Wherein, each of the at least two candidate region images contains a candidate annotated region for annotating abnormal objects in the original image (e.g., the above-mentioned original image T). Wherein, the number of original images can be at least two, and each of the at least two original images corresponds to at least two candidate region images. The following steps S1022-step S1023 are described by taking the at least two candidate region images as candidate region images corresponding to the original image T as an example.

[0164] It can be understood that the number of the at least two annotated objects can be the same as the number of the at least two candidate region images, or the number of the at least two annotated objects can be different from the number of the at least two candidate region images, which is not limited here. Therefore, the number of the at least two candidate region images corresponding to each original image can be the same or different. The number of the at least two candidate region images is not limited here.

[0165] It is understandable that the annotation object here may be the first annotation object associated with the first annotation terminal, and the annotation object associated with the second annotation terminal may be the second annotation object. The first annotation object and the second annotation object may be the same annotation object, or the first annotation object and the second annotation object may not be the same annotation object. It should be understood that the embodiment of the present application does not limit the number of first annotation objects, and the embodiment of the present application does not limit the number of second annotation objects.

[0166] Step S1022, determining the region review result according to the candidate annotated regions included in each candidate region image and the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image;

[0167] Specifically, the first annotation terminal can determine the number of candidate areas corresponding to the candidate annotation areas included in each candidate area image, compare the number of at least two candidate areas, and obtain a number comparison result. Further, if the number comparison result indicates that there are at least S identical candidate areas in at least two candidate areas, the first annotation terminal can determine at least S identical candidate areas as the target candidate areas, and determine the area review result based on the candidate area images associated with the target candidate areas. Here, S here can be a positive integer that is less than or equal to the number of images of at least two candidate area images. Optionally, if the number comparison result indicates that there are not at least S identical candidate areas in at least two candidate areas, the first annotation terminal can determine the candidate annotation areas in at least two candidate area images as the area review result.

[0168] It can be understood that S here is determined by the number of at least two candidate regions (i.e., the number of at least two candidate region images). For example, when the number of at least two candidate region images is equal to 4, the 4 candidate region images may include candidate region images X 1 , candidate region image X 2 , candidate region image X 3 and the candidate region image X 4 Among them, the candidate region image X 1 The number of candidate regions corresponding to the candidate annotation regions included may be O 1 , candidate region image X 2 The number of candidate regions corresponding to the candidate annotation regions included may be O 2 , candidate region image X 3 The number of candidate regions corresponding to the candidate annotation regions included may be O 3 , candidate region image X 4 The number of candidate regions corresponding to the candidate annotation regions included may be O 4 Therefore, when the number of images of at least two candidate region images is equal to 4, S may be equal to 2.

[0169] Among them, the number of candidate regions O 1 , the number of candidate regions O 2 and the number of candidate regions O 3 The first annotation terminal can be the same as the number of candidate regions O 1 , the number of candidate regions O 2 , the number of candidate regions O 3 and the number of candidate regions O 4A quantity comparison is performed to obtain a first quantity comparison result, which indicates that there are at least S (in this case, 3) identical candidate region numbers (i.e., the candidate region number O) in at least two candidate region numbers. 1 , the number of candidate regions O 2 and the number of candidate regions O 3 ). Further, the first annotation terminal may determine the number of the three identical candidate regions as the number of target candidate regions (ie, the number of target candidate regions is equal to 0). 1 ), and then according to the candidate region image associated with the number of target candidate regions (i.e., candidate region image X 1 , candidate region image X 2 and the candidate region image X 3 ) to determine the regional audit results.

[0170] Optionally, the number of candidate regions O 1 , the number of candidate regions O 2 , the number of candidate regions O 3 and the number of candidate regions O 4 The first marking terminal can be different from the candidate area number O 1 , the number of candidate regions O 2 , the number of candidate regions O 3 and the number of candidate regions O 4 The first annotation terminal may compare the at least two candidate region images (i.e., the candidate region images X and X) to obtain a second comparison result, wherein the second comparison result indicates that there are not at least S (in this case, 1) identical candidate region numbers in the at least two candidate region numbers. 1 , candidate region image X 2 , candidate region image X 3 and the candidate region image X 4 ) are determined as the regional review results.

[0171] It should be understood that the specific process of the first annotation terminal determining the regional audit result based on the candidate area images associated with the number of target candidate areas can be described as follows: the first annotation terminal can obtain the candidate annotation areas included in every two candidate area images in the target candidate area image, and determine the second area overlap between the candidate annotation areas included in every two candidate area images. Among them, the target candidate area image is a candidate area image associated with the number of target candidate areas. Further, the first annotation terminal can compare the second area overlap with the second overlap threshold value to obtain the overlap comparison result. Further, if the overlap comparison result indicates that in the target candidate area image, there are at least P candidate area images in which the second area overlap corresponding to every two candidate area images is greater than the second overlap threshold, then the first annotation terminal can merge the candidate annotation areas included in at least P candidate area images to obtain a merged candidate area, and determine the merged candidate area as the regional audit result. Among them, P here can be a positive integer less than or equal to the number of images of the target candidate area image. Optionally, if the overlap comparison result indicates that in the target candidate area image, there is no second area overlap corresponding to every two candidate area images among at least P candidate area images that is greater than a second overlap threshold, then the first annotation terminal may determine the candidate annotation area included in the target candidate area image as the area review result.

[0172] Among them, the target candidate area image includes the candidate area image X q and the candidate region image X w , candidate region image X q Including candidate annotation area A v , candidate region image X w Including candidate annotation area B z , where q and w are both positive integers less than or equal to the number of target candidate region images, q is not equal to w, where v and z are both less than or equal to the number of target candidate regions. It should be understood that the specific process of the first annotation terminal determining the second region overlap between the candidate annotation regions included in each two candidate region images can be described as follows: the first annotation terminal can determine the candidate annotation region A v With the candidate region image X w Included candidate annotation area B z The first candidate overlap between them, the maximum candidate overlap among the first candidate overlaps is determined as the candidate annotation area A v Further, the first annotation terminal can determine the candidate annotation area B z With the candidate region image X q Included candidate annotation area A vThe second candidate overlap between them, the maximum candidate overlap among the second candidate overlaps is determined as the candidate annotation area B z Further, the first annotation terminal can mark the candidate region image X q The first coincidence degree of each candidate annotation region in the candidate region image X w The second coincidence degree of each candidate annotation region in is determined as the candidate region image X q and the candidate region image X w The second area overlap between them.

[0173] It can be understood that if the candidate region image X q The first coincidence degree of each candidate annotation region in the candidate region image X w If the second coincidence degree of each candidate annotation region in is greater than the second coincidence degree threshold, it indicates that the candidate region image X q and the candidate region image X w The second region overlap between the target candidate region images is greater than the second overlap threshold. The first annotation terminal may determine the overlap between the candidate annotation regions in different target candidate region images by using the above formula (1).

[0174] It can be understood that the first marking terminal can mark the candidate marking area A v The first coincidence degree in the candidate region image X w The first matching annotation area determined by the candidate annotation area corresponding to , that is, the candidate annotation area A v and the first matching marked area are mutually matched marked areas; the first marking terminal can also mark the candidate marked area B z The second coincidence degree in the candidate region image X q The candidate annotation area corresponding to is determined as the second matching annotation area, that is, candidate annotation area B z The first matching marked area and the second matching marked area are mutually matching marked areas. The first matching marked area and the second matching marked area can be collectively referred to as matching marked areas.

[0175] It can be understood that when the first annotation terminal merges the candidate annotation regions included in the at least P candidate region images, the candidate annotation regions that match each other in the at least P candidate region images can be merged to obtain the merged candidate regions corresponding to the at least P candidate region images. For example, the at least P candidate region images can include the candidate region image X 11 and the candidate region image X 22 , candidate region image X 11 Can include candidate annotation area A 1 and candidate annotation region A 2 , candidate region image X22 Can include candidate annotation area B 1 and candidate annotation region B 2 Candidate annotation area A 1 and candidate annotation region B 1 Mutually matching annotation regions, candidate annotation regions A 2 and candidate annotation region B 2 In this way, the first marking terminal can mark the candidate area A 1 and candidate annotation region B 1 Merge and obtain the merged candidate region D 1 , and the candidate annotation area A 2 and candidate annotation region B 2 Merge and obtain the merged candidate region D 2 , and then merge the candidate region D 1 and merge candidate regions D 2 Determined as a regional audit result.

[0176] Optionally, the first annotation terminal may also obtain an auxiliary candidate region image from the at least P candidate region images, and determine the auxiliary candidate region image as the region audit result. The auxiliary candidate region image may be any one of the at least P candidate region images. For example, the at least P candidate region images include the candidate region image X 11 and the candidate region image X 22 When the first annotation terminal can 11 As auxiliary candidate region image.

[0177] Optionally, the first annotation terminal may also select any candidate annotation region from each mutually matching annotation region of at least P candidate region images to obtain a candidate annotation region set, and determine the candidate annotation region set as the region review result. For example, the at least P candidate region images include the candidate region image X 11 and the candidate region image X 22 When the first annotation terminal can 11 Candidate annotation region A in 1 and the candidate region image X 22 Candidate annotation region B in 2 , determined as a candidate annotation region set, the first annotation terminal may also convert the candidate region image X 11 Candidate annotation region A in 1 and candidate annotation region A 2 , determined as a set of candidate annotation regions.

[0178] It can be understood that P here is determined by the number of target candidate region images. For example, when the number of target candidate region images is 3, the three target candidate region images may include candidate region images X 1 , candidate region image X 2 and the candidate region image X 3 , here the candidate region image X 1 , candidate region image X 2 and the candidate region image X 3 The number of candidate annotation regions included in each of the candidate regions is 1. 1 Can include candidate annotation region R 1 , candidate region image X 2 Can include candidate annotation region R 2 , candidate region image X 3 Can include candidate annotation region R 3 Candidate annotation region R 1 and the candidate annotation region R 2 The second area overlap between the two regions can be the area overlap C 1 , candidate annotation region R 1 and the candidate annotation region R 3 The second area overlap between the two regions can be the area overlap C 2 , candidate annotation region R 2 and the candidate annotation region R 3 The second area overlap between the two regions can be the area overlap C 3 Therefore, when the number of target candidate region images is equal to 3, P may be equal to 2.

[0179] Among them, the regional overlap C 1 can be greater than a second overlap threshold (eg, 0.3), the region overlap C 2 and regional overlap C 3 The first annotation terminal may set the region overlap degree C 1 、Regional overlap C 2 and regional overlap C 3 The first coincidence comparison result is obtained by comparing the coincidence with the second coincidence threshold, and the first coincidence comparison result indicates that there are at least P (in this case, there are 2) candidate area images in the target candidate area image, and the second area coincidence (i.e., the area coincidence C) corresponding to every two candidate area images is 1 ) is greater than the second coincidence threshold. Further, the first annotation terminal may mark at least P candidate region images (ie, candidate region images X 1 and the candidate region image X 2) are merged to obtain a merged candidate region.

[0180] Optional, where the area overlap C 1 、Regional overlap C 2 and regional overlap C 3 The first annotation terminal can set the region overlap degree C 1 、Regional overlap C 2 and regional overlap C 3 The first annotation terminal may compare the target candidate region image with the second annotation region (i.e., the candidate annotation region R ) included in the target candidate region image. 1 , candidate annotation region R 2 and the candidate annotation region R 3 ) is determined as the regional audit result.

[0181] Among them, the first annotation terminal is for the candidate area image X 1 and the candidate region image X 2 The specific process of merging the candidate annotation areas included can be found in Figure 7 , Figure 7 is a schematic diagram of a scenario for determining a candidate region for merging provided in an embodiment of the present application. Figure 7 The user terminal 70a shown may be a first annotation terminal, such as Figure 7 The image 70b and the image 70c may be candidate region images corresponding to the original image (eg, the original image T). Here, the candidate region image 70b may be the candidate region image X. 1 , where the candidate region image 70c may be the candidate region image X 2 The candidate region image 70b may include the candidate annotation region 71a, and the candidate region image 70c may include the candidate annotation region 71b.

[0182] like Figure 7As shown, the first annotation terminal can merge the candidate annotation area 71a and the candidate annotation area 71b to obtain a merged candidate area, which can be the merged candidate area 71c in the image 70d, and then determine the merged candidate area 71c as the regional review result. Optionally, the first annotation terminal can also directly determine the candidate area image 70b or the candidate area image 70c as the regional review result. Optionally, the first annotation terminal can also determine the candidate annotation area 71a or the candidate annotation area 71b as the regional review result.

[0183] Optionally, when the candidate area image 70b also includes a candidate annotation area 71d (not shown in the figure) and the candidate area image 70c also includes a candidate annotation area 71e (not shown in the figure), the first annotation terminal may also merge the candidate annotation area 71d (not shown in the figure) and the candidate annotation area 71e (not shown in the figure) to obtain a merged candidate area, which may be a merged candidate area 71f (not shown in the figure), and then determine the merged candidate area 71f (not shown in the figure) as the area review result.

[0184] Step S1023, determining a standard marking area for marking abnormal objects according to the area review result.

[0185] Specifically, if the regional review result is a candidate annotation area in at least two candidate area images, the first annotation terminal may send the regional review result to the first audit terminal, so that the first audit terminal determines the first candidate review area to be sent to the second audit terminal based on the candidate annotation area in at least two candidate area images. The first candidate review area is used to instruct the second audit terminal to return a standard annotation area for marking abnormal objects. Optionally, if the regional review result is a candidate annotation area included in the target candidate area image, the first annotation terminal may send the regional review result to the first audit terminal, so that the first audit terminal determines the second candidate review area to be sent to the second audit terminal based on the candidate annotation area included in the target candidate area image. The second candidate review area is used to instruct the second audit terminal to return a standard annotation area for marking abnormal objects. Optionally, if the regional review result is a merged candidate area, the first annotation terminal may send the regional review result to the second audit terminal, so that the second audit terminal returns a standard annotation area for marking abnormal objects based on the merged candidate area.

[0186] It is understandable that the first audit terminal may include one or more audit terminals, and there is no limit on the number of first audit terminals; the second audit terminal may include one or more audit terminals, and there is no limit on the number of second audit terminals.

[0187] It is understandable that the level of the second review user can be higher than that of the first review user, and the level of the first review user can be higher than that of the annotation object. Among them, when the original image is a medical image, at least two annotation objects can be doctors with at least 3 years of operation experience, and at the same time meet the requirements of the registration unit being a second-level or third-level hospital, a doctor or above, and pass the training and assessment related to the annotation doctor of the annotation demand unit. Correspondingly, the review user corresponding to the first review terminal can be the first review user, and the first review user (i.e., arbitration expert) can be a doctor with at least 10 years of operation experience, and at the same time meet the requirements of the registration unit being a third-level A-class hospital, a deputy director or above, and pass the training and assessment related to the arbitration expert of the annotation demand unit. Correspondingly, the review user corresponding to the second review terminal can be the second review user, and the second review user (i.e., audit expert) can be a doctor with at least 10 years of operation experience, and at the same time meet the requirements of the registration unit being a third-level A-class hospital, a chief physician or above, and pass the training and assessment related to the audit expert of the annotation demand unit.

[0188] It is understandable that when the second annotation terminal determines the standard annotation area for annotating abnormal objects in the second original image, the area review result corresponding to the second annotation terminal can be sent to the first review terminal or the second review terminal corresponding to the second annotation terminal. The first review terminal corresponding to the second annotation terminal can be the same review terminal as the above-mentioned first review terminal, or a different review terminal; the second review terminal corresponding to the second annotation terminal can be the same review terminal as the above-mentioned second review terminal, or a different review terminal. Among them, under the same task, the above-mentioned review terminals are the same. For example, the task here can be to assign 1,000 images to 50 doctors for labeling (here it is assumed that one doctor corresponds to one annotation terminal). Doctor Y 1 100 images can be allocated, ..., Doctor Y 50 20 images can be allocated, so that doctor Y 1 and Doctor Y 2 The first audit terminal and the second audit terminal corresponding to each other may be the same.

[0189] It is understandable that in order to ensure the quality of data annotation and reduce individual differences between doctors, the annotation process can be multi-doctor annotation. The multi-doctor annotation process can include multiple doctors' independent annotation, arbitration and review (quality control) and other links. The annotation results before and after the review (quality control) are the candidate annotation gold standard and the annotation gold standard respectively. Among them, the process of multiple doctors independently outlining and annotating can be seen in Figure 8a , Figure 8a FIG. 1 is a flow chart of a method for multiple doctors to determine a standard region image provided by an embodiment of the present application. Figure 8aThe image shown may be the original image T mentioned above, and the original image T may be annotated by an artificial intelligence model to obtain an auxiliary annotation result corresponding to the original image T (i.e., a predicted region image corresponding to the predicted annotated region). The original image T may be a medical image.

[0190] like Figure 8a As shown, m annotation doctors can independently outline and annotate the original image T based on the auxiliary annotation results to confirm whether the medical image contains a lesion. If the medical image contains a lesion, the m annotation doctors annotate the location and shape of the lesion in the form of a polygon / region, and obtain the annotation results corresponding to each of the m annotation doctors (i.e., the candidate region image corresponding to the candidate annotation region). Here, m can be a positive integer greater than 1, and the m annotation doctors can specifically include doctor Y 1 , ..., Doctor Y m , Doctor Y 1 , ..., Doctor Y m Multiple doctors on the first labeling terminal can be marked. Among them, doctor Y 1 The corresponding annotation result can be the annotation result X 1 ,…,Doctor Y m The corresponding annotation result can be the annotation result X m .

[0191] It should be understood that when annotating, the annotating doctor should be as close to the edge of the lesion as possible and outline the entire lesion within the ROI (Region of Interest). The present application embodiment does not limit the annotation form of the above polygon / region. The annotated area can be called the ROI area.

[0192] like Figure 8a As shown, the first annotation terminal obtains the above annotation result X 1 ,…, labeling result X m Afterwards, the number (i.e., the number) of candidate annotation regions included in each annotation result may be determined. If the number of candidate annotation regions included in each annotation result is the same, pairing calculation may be performed on the candidate annotation regions included in each annotation result. Further, when performing pairing calculation on the candidate annotation regions included in each annotation result, the first annotation terminal may determine the pairing coincidence (i.e., the second region coincidence) between the candidate annotation regions included in each annotation result. Then, when any pairing coincidence is greater than the coincidence threshold (i.e., the second coincidence threshold), based on the annotation result X, the first annotation terminal may determine the pairing coincidence (i.e., the second region coincidence) between the candidate annotation regions included in each annotation result. 1 ,…, labeling result X m, and obtain the candidate annotation gold standard. Among them, the IOU area is obtained by comparing the ROIs outlined by each doctor in pairs. When there is at least one annotation result ROI that cannot find an ROI with an IOU (i.e., pairing overlap) greater than the overlap threshold (e.g., 0.3) in the other annotation result, the two annotation results are considered inconsistent. Furthermore, the first annotation terminal can send the candidate annotation gold standard to the review expert, so that the review expert reviews the candidate annotation gold standard and obtains the annotation gold standard (i.e., the standard area image corresponding to the standard annotation area).

[0193] At this time, the number of ROIs obtained by multiple doctors' independent labeling is consistent, and the ROIs outlined by each doctor are compared with the IOU areas. When any labeled ROI area has another labeled ROI with an IOU greater than the overlap threshold, it is considered that the two ROIs are annotated with the same lesion area, and the two ROIs agree on the boundary height (that is, the area overlaps more), then any ROI that satisfies the IOU greater than the overlap threshold can be taken as the labeling result of the lesion, or the two ROIs can be merged to obtain the labeling result.

[0194] Optional, such as Figure 8a As shown, if the number of candidate annotation regions included in each annotation result is inconsistent, the first annotation terminal can send the m annotation results to the arbitration expert, so that the arbitration expert can review each annotation result to obtain the first candidate review region (i.e., the candidate annotation gold standard). Further, the arbitration expert can send the first candidate review region to the review expert, and the review expert can review the first candidate review region to obtain the annotation gold standard.

[0195] Optional, such as Figure 8a As shown, if there is no pairing whose coincidence is greater than the coincidence threshold, the first annotation terminal can send the m annotation results to the arbitration expert, so that the arbitration expert can review each annotation result to obtain the second candidate review area (i.e., the candidate annotation gold standard). Further, the arbitration expert can send the second candidate review area to the review expert, and the review expert can review the second candidate review area to obtain the annotation gold standard.

[0196] Among them, the arbitration expert can select or re-annotate based on the m annotation results to obtain the arbitration result. At this time, the candidate annotation gold standard shall be based on the arbitration result.

[0197] Among them, the review experts can perform quality control on the candidate gold standard annotations one by one. Those that pass the quality control are called the gold standard annotations, and those that fail the quality control are sent back to the independent annotation or arbitration link, and the annotation task is re-performed from the link that was sent back. Among them, it can be understood that the doctor corresponding to the independent annotation here can be the above-mentioned doctor Y 1, ..., Doctor Y m , or other doctors, which is not limited in this application.

[0198] Optionally, the number of the annotation object may be one, and when the number of the annotation object is one, the first annotation terminal may obtain a candidate region image provided by the one annotation object for the original image. The one candidate region image includes a candidate annotation region for annotating an abnormal object in the original image. Further, the first annotation terminal may use the candidate annotation region included in the one candidate region image as a standard annotation region for annotating an abnormal object.

[0199] The process of a single doctor performing independent outlining and annotation can be found in Figure 8b , Figure 8b FIG. 1 is a flow chart of a single doctor determining a standard region image provided by an embodiment of the present application. Figure 8b The image shown can be the above Figure 8a The original image T in the corresponding embodiment can be assisted in annotation by the artificial intelligence model to obtain an assisted annotation result corresponding to the original image T (ie, a predicted area image corresponding to the predicted annotation area).

[0200] like Figure 8b As shown, the number of annotation doctors can be one, and the one annotation doctor can outline and annotate the original image T based on the auxiliary annotation result to obtain the annotation result corresponding to the one doctor (i.e., the candidate region image corresponding to the candidate annotation region). Furthermore, the first annotation terminal can directly use the annotation result as the annotation gold standard (i.e., the standard region image corresponding to the standard annotation region).

[0201] It is understandable that the application of this solution on the product side is mainly front-end interaction and back-end processing. The business logic can process the above medical images through the artificial intelligence model of auxiliary annotation of the back-end to obtain auxiliary annotation results, which can be displayed on the front end together with the medical images. In this way, the annotator can annotate the medical images according to the auxiliary annotation results and finally obtain the annotation results of the medical data. In addition, the annotator can use the annotated data to train / tune the artificial intelligence model of auxiliary annotation, so that the model has better generalization performance or is more in line with personal annotation habits.

[0202] It can be seen that the embodiment of the present application can generate at least two candidate area images based on the original image, and then determine the standard annotation area for annotating abnormal objects in the original image according to the candidate annotation area included in each of the at least two candidate area images, and the number of candidate areas corresponding to the candidate annotation area included in each candidate area image. The above process can make up for the problem of insufficient individual ability or ability difference of a single annotation person (i.e., the annotation object), so it can improve the accuracy of annotating abnormal objects in the original image.

[0203] For further information, see Fig. 9 , Fig. 9 is a flowchart of an image data processing method provided by an embodiment of the present application. The method can be executed by a server, or by a first annotation terminal, or by both the server and the first annotation terminal. The server can be the above-mentioned Figure 2 In the corresponding implementation of the server 20a, the first annotation terminal may be the above Figure 2 The corresponding user terminal 20b in the embodiment. For ease of understanding, this embodiment takes the method being performed by the server and the first annotation terminal as an example for explanation. The image data processing method may include the following steps S201-S208:

[0204] Step S201, the server sends the initial network model to the first annotation terminal and the second annotation terminal;

[0205] Among them, the initial network model can be used to mark the regions of abnormal objects.

[0206] Step S202, the first annotation terminal obtains an initial network model from the server, and annotates the original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the original image;

[0207] The original image here may be a first original image, and the predicted annotation region here may be a first predicted annotation region.

[0208] It can be understood that the labeling terminal may include but is not limited to a first labeling terminal. When the labeling terminal also includes a second labeling terminal, the second labeling terminal can obtain an initial network model from a server, perform region labeling on the second original image based on the initial network model, and obtain a second predicted labeling region for labeling abnormal objects in the second original image.

[0209] The specific process of the first annotation terminal annotating the original image region based on the initial network model can be found in the above Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0210] In recent years, with the breakthrough of a new generation of artificial intelligence technology represented by deep learning, revolutionary progress has been made in the field of automatic recognition of medical images. Artificial intelligence in real-time assisted detection and classification of lesions in medical images is expected to help clinicians improve the quality of examinations and reduce missed lesions.

[0211] Excellent computer-aided diagnosis (CAD) performance relies on a large amount of representative high-quality annotated data to fully learn the characteristics of lesions. The quality of data annotation determines the stability and accuracy of the algorithm model. However, due to the obvious individual differences and complexity of various modal data and different disease lesions, annotation is extremely difficult. Effectively collecting large samples of high-quality annotated data has become the primary basis and prerequisite for CAD development.

[0212] Optionally, the initial network model in the embodiment of the present application can also be used to classify the original image, and the initial network model here can be a segmentation model. The segmentation model is a mathematical model obtained by machine learning technology after learning the labeled sample data (image-category correspondence), and the parameters of the mathematical model can be obtained during the learning and training process, and the parameters of the mathematical model are loaded during recognition prediction to calculate and generate the segmentation result of the input sample (i.e., the original image) belonging to the lesion part (for example, the lung nodule part).

[0213] Therefore, the initial network model can perform local segmentation on the original image, group together things belonging to the same category in the image, and distinguish each sub-part, and then perform simple classification on the original image, determine the category of each sub-part, and thus determine the image category of the image. Among them, the image category can represent the category to which the image content belongs, that is, benign or malignant. Among them, sensitivity and specificity can be used to determine the performance of the segmentation model. Sensitivity can represent the proportion of images detected as malignant among all malignant (i.e., lung cancer) images. Specificity can represent the proportion of images detected as benign among all benign (i.e., no lung cancer) images.

[0214] Step S203, the first annotation terminal obtains a candidate annotation area provided by the annotation object for annotating an abnormal object in the original image, performs an area review on the candidate annotation area, and obtains a standard annotation area for annotating the abnormal object;

[0215] The specific process of the first annotation terminal performing regional review on the candidate annotation area can be referred to in the above Figure 6 The description of step S1021 to step S1023 in the corresponding embodiment will not be repeated here.

[0216] Step S204: the first annotation terminal adjusts the model parameters of the initial network model based on the predicted annotation area and the standard annotation area to obtain first updated model parameters, and sends the first updated model parameters to the server;

[0217] The specific process of adjusting the model parameters of the initial network model by the first annotation terminal based on the predicted annotation area and the standard annotation area can be referred to in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0218] Step S205, the server receives the first updated model parameter returned by the first annotation terminal;

[0219] The first updated model parameters are obtained by adjusting the model parameters of the initial network model by the first annotation terminal based on the predicted annotation area and the standard annotation area for annotating abnormal objects. The standard annotation area is obtained by performing an area review on the candidate annotation area after obtaining the candidate annotation area provided by the annotation object for annotating abnormal objects in the original image. Optionally, when receiving the first updated model parameters sent by the first annotation terminal, the server can also receive the second updated model parameters returned by the second annotation terminal.

[0220] It should be understood that the specific process of the second annotation terminal determining the second update model parameters can refer to the description of the first annotation terminal determining the first update model parameters, which will not be repeated here.

[0221] It can be understood that the first annotation terminal can obtain the public key of the server, encrypt the first update model parameter by the public key of the server, obtain the encrypted first update model parameter, and then send the encrypted first update model parameter to the server. In this way, after receiving the encrypted first update model parameter sent by the first annotation terminal, the server can decrypt the encrypted first update model parameter by the private key of the server to obtain the first update model parameter. Optionally, the first annotation terminal can also encrypt the encrypted first update model parameter by the private key of the first annotation terminal to obtain the encrypted model parameter, and then send the encrypted model parameter to the server. In this way, after receiving the encrypted model parameter, the server can obtain the public key of the first annotation terminal, decrypt the encrypted model parameter by the public key of the first annotation terminal, and obtain the encrypted first update model parameter.

[0222] It is understandable that the server can obtain the updated model parameters received in the target period, and then perform the following step S206 on the updated model parameters received in the target period. For example, the updated model parameters received by the server in the target period can be the first updated model parameters and the second updated model parameters.

[0223] The target period may be in hours, minutes, or seconds, and the unit of the target period is not limited here. For example, when the target period is in hours, the target period may be 1 hour; for another example, when the target period is in minutes, the target period may be 1 minute; for another example, when the target period is in seconds, the target period may be 30 seconds.

[0224] Optionally, when the annotation terminal includes but is not limited to the above-mentioned first annotation terminal and the second annotation terminal, the updated model parameters received by the server include but are not limited to the above-mentioned first updated model parameters and the second updated model parameters, and can also receive other updated model parameters returned by other annotation terminals, and then execute the following step S206 based on the received updated model parameters.

[0225] Step S206, the server performs parameter fusion on the first updated model parameters and the second updated model parameters to generate target model parameters;

[0226] It is understandable that the server can perform parameter fusion of the first update model parameter and the second update model parameter by weighted summation. The server can determine the weights of the first annotation terminal and the second annotation terminal according to the importance of the first annotation terminal and the second annotation terminal. For example, the importance here can be seniority. When the seniority of the first annotation object corresponding to the first annotation terminal is greater than the seniority of the second annotation object corresponding to the second annotation terminal, the weight of the first annotation terminal can be greater than the weight of the second annotation terminal.

[0227] Optionally, if the update model parameter received by the server within the target period is the first update model parameter, the server may directly use the first update model parameter as the target model parameter. Optionally, if the update model parameter received by the server within the target period is the second update model parameter, the server may directly use the second update model parameter as the target model parameter.

[0228] Step S207, the server sends the target model parameters to the first annotation terminal;

[0229] The server may also send the target model parameters to the second annotation terminal.

[0230] Step S208: The first annotation terminal receives the target model parameters returned by the server, updates the initial network model using the target model parameters, and obtains a target network model for annotating the region of the abnormal object.

[0231] It is understandable that the first annotation terminal can update the initial network model in the first annotation terminal through the received target model parameters to obtain the target network model for regional annotation of abnormal objects; the second annotation terminal can also update the initial network model in the second annotation terminal through the received target model parameters to obtain the target network model for regional annotation of abnormal objects. At this time, the models on the first annotation terminal and the second annotation terminal are the same, both of which are target network models.

[0232] For easier understanding, see Fig.10 , Fig.10 Schematic diagram of a flow chart of model training provided in the embodiment of the present application. Fig.10 The training process of a single batch of data is shown. Model 100a may be a remote model deployed on a server. The number of annotation terminals (i.e., participants) corresponding to the server may be N, where N may be a positive integer. The N participants may download the latest remote model (i.e., initial network model) from the server. The N parameter parties may be participants Y, Y, and Y, respectively. 1 、Participant Y 2 , …, Participant Y N .

[0233] like Fig.10 The model 100b shown can be for participant Y 1 The initial network model in Fig.10 The model 100c shown can be for participant Y 2 The initial network model in , …, such as Fig.10 The model 100d shown can be for participant Y N At this time, the model 100b, the model 100c and the model 100d can be the same network model (for example, the model G 1 ).

[0234] like Fig.10 The data shown can be a raw image, such as Fig.10 The annotation result shown can be the annotation gold standard (i.e., the standard region image); Fig.10 The data settings shown may be predefined rules (i.e., model update conditions) set by different annotation terminals, and different annotation terminals may correspond to different model update conditions. 1The data and the annotation results (i.e., the first original data and the standard region image corresponding to the first original data) in the data and the annotation results E 1 , Participant Y 2 The data and the annotation result in (i.e., the second original data and the standard region image corresponding to the second original data) can be the data and the annotation result E 2 ,…,Participant Y N The data and the annotation results (i.e., the third original data and the standard region image corresponding to the third original data) in the data and the annotation results E N Among them, participant Y 1 The corresponding data setting can be data setting K 1 , Participant Y 2 The corresponding data setting can be data setting K 2 ,…,Participant Y N The corresponding data setting can be data setting K N .

[0235] like Fig.10 As shown in the figure, each participant can use the model to train the local data that meets the settings according to the data settings. Each participant (i.e., the annotator) can increase the role of model builder from model user, thereby improving the effectiveness of auxiliary annotation. 1 K can be set according to the data 1 , set K to match the data 1 Data and annotation results E 1 Using model 100b for training; Participant Y 2 K can be set according to the data 2 , set K to match the data 2 Data and annotation results E 2 Using model 100c for training; ...; Party Y N K can be set according to the data N , set K to match the data N Data and annotation results E N The model was trained using 100d.

[0236] Among them, it can be understood that model training can obtain gradient information (i.e., update model parameters), and each participant can encrypt the gradient information and upload the encrypted gradient information to the server so that the server aggregates the gradient information of all participants to obtain update information, which is used to update the parameters of the remote model.

[0237] like Fig.10 As shown, participant Y 1 、Participant Y 2 , …, Participant Y NThe encrypted gradient information can be uploaded through the solid line, and the server can return the updated model update parameters (i.e., the model update information) to each participant through the dotted line. In this way, each participant can update their own model after receiving the update information. 1 Model 100b can be updated by updating information, and participant Y 2 Model 100c can be updated by updating information, ..., participant Y N The model 100d may be updated by the update information.

[0238] Optionally, it is understood that, in general, Party Y 1 、Participant Y 2 , …, Participant Y N The gradient information will not be returned at the same time in the target period, that is, in the target period, the participant Y 1 、Participant Y 2 , …, Participant Y N There are participants who do not meet the above data settings. Participants who do not meet the data settings will not return gradient information. For example, the server can receive participant Y in the target period. 1 and participant Y 2 The returned gradient information, therefore, can be used by the server to 1 Corresponding gradient information and participant Y 2 The corresponding gradient information is aggregated to obtain the updated information, and then the updated information is returned to the participant Y 1 and participant Y 2 At this time, the server does not need to return the updated information to any other party except participant Y. 1 and participant Y 2 This means that other participants have no need to update the initial network model.

[0239] Among them, it is understandable that the above-mentioned participant Y 1 、Participant Y 2 , …, Participant Y N The models in can be the same or different. For example, in T 1 At time Y 1 The model 100b in the example may be model G 1 , Participant Y 2 The model 100c in the example may be model G 1 ,…,Participant Y N The model 100d in the example can be model G 1 For example, in T 2 At time Y 1 The model 100b in the example may be model G 2 , Participant Y2 The model 100c in the example may be model G 2 ,…,Participant Y N The model 100d in the example can be model G 1 Therefore, the initial network models corresponding to model training by different participants are different. For example, in T 3 At time Y 1 The initial network model in can be model G 2 , Participant Y 2 The initial network model in can be model G 2 ,…,Participant Y N The initial network model in can be model G 1 Among them, T 2 The time can be T 1 The next moment of the moment, T 3 The time can be T 2 The next moment of the moment.

[0240] It can be seen that the key to the auxiliary annotation method based on federated learning provided by the present application is that the update of the model uses the newly annotated data of multiple annotated objects (e.g., doctors) in the system, and the training of the model uses federated learning, thereby avoiding the risk of data leakage caused by the data being transmitted back from the annotated object side to the model side, and protecting the privacy of the data. In addition, the annotated object can configure its own annotated data for the model training part, that is, adjust the parameters of the model. It can be understood that federated learning can be regarded as sample-based distributed model training, and each machine (i.e., the annotation terminal) can download the model (i.e., the initial network model) from the server, and then use their own data to train the model, and then return to the server the parameters that need to be updated (i.e., update the model parameters). The server aggregates the parameters returned on each machine, updates the model, and then feeds back the latest model to each machine, that is, the target model parameters of the updated model are transmitted back to the machine, so that the accuracy of the model for auxiliary annotation can be improved.

[0241] For further information, see Fig.11 , Fig.11 1 is a schematic diagram of the structure of an image data processing device provided in an embodiment of the present application. The image data processing device 1 may include: a model acquisition module 11, a region review module 12, a first sending module 13, and a first receiving module 14;

[0242] The model acquisition module 11 is used for the first annotation terminal to obtain an initial network model from a server, and to annotate the original image based on the initial network model to obtain a predicted annotation area for annotating abnormal objects in the original image;

[0243] The model acquisition module 11 includes: a feature extraction unit 111, a position determination unit 112, and a region labeling unit 113;

[0244] A feature extraction unit 111 is used for the first annotation terminal to obtain an initial network model from a server, input the original image into the initial network model, and extract an image feature vector of the original image through the initial network model;

[0245] A position determination unit 112, configured to determine, based on the image feature vector, the predicted region position information associated with the abnormal object in the original image;

[0246] The region labeling unit 113 is used to label the region of the original image according to the predicted region position information, and generate a predicted labeling region for labeling abnormal objects.

[0247] The specific implementation of the feature extraction unit 111, the position determination unit 112 and the region labeling unit 113 can be found in the above Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0248] A region review module 12 is used to obtain a candidate marked region provided by the marked object for marking an abnormal object in the original image, perform a region review on the candidate marked region, and obtain a standard marked region for marking the abnormal object;

[0249] Wherein, the number of the annotated objects is at least two;

[0250] The area audit module 12 includes: an image acquisition unit 121, an area audit unit 122, and an area determination unit 123;

[0251] The image acquisition unit 121 is used to acquire at least two candidate region images provided by at least two annotated objects; each of the at least two candidate region images includes a candidate annotated region for annotating an abnormal object in the original image;

[0252] The region review unit 122 is used to determine the region review result according to the candidate annotated regions included in each candidate region image and the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image;

[0253] The area review unit 122 includes: a quantity comparison subunit 1221, a first determination subunit 1222, and a second determination subunit 1223;

[0254] The quantity comparison subunit 1221 is used to determine the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image, and to compare the numbers of at least two candidate regions to obtain a quantity comparison result;

[0255] The first determination subunit 1222 is used to determine the at least S identical candidate region numbers as the target candidate region number if the number comparison result indicates that there are at least S identical candidate region numbers in the at least two candidate region numbers, and determine the region audit result according to the candidate region images associated with the target candidate region number; S is a positive integer less than or equal to the number of images of the at least two candidate region images;

[0256] The first determination subunit 1222 is specifically used to obtain the candidate annotation regions included in every two candidate region images in the target candidate region image, and determine the second region overlap between the candidate annotation regions included in every two candidate region images; the target candidate region image is a candidate region image associated with the number of target candidate regions;

[0257] The first determining subunit 1222 is further specifically used to compare the second region coincidence with the second coincidence threshold to obtain a coincidence comparison result;

[0258] The first determination subunit 1222 is further specifically used to merge the candidate annotation areas included in at least P candidate area images to obtain a merged candidate area, and determine the merged candidate area as the area review result if the overlap comparison result indicates that in the target candidate area image, there are at least P candidate area images whose second area overlap corresponding to every two candidate area images is greater than a second overlap threshold; P is a positive integer less than or equal to the number of images of the target candidate area image;

[0259] The first determination subunit 1222 is also specifically used to determine the candidate annotation area included in the target candidate area image as the area review result if the overlap comparison result indicates that in the target candidate area image, there is no second area overlap corresponding to every two candidate area images in at least P candidate area images that is greater than a second overlap threshold.

[0260] The second determination subunit 1223 is used to determine the candidate annotated areas in at least two candidate area images as area review results if the quantity comparison result indicates that there are not at least S identical candidate area quantities in at least two candidate area quantities.

[0261] The specific implementation of the quantity comparison subunit 1221, the first determination subunit 1222 and the second determination subunit 1223 can be found in the above Figure 6 The description of step S1022 in the corresponding embodiment will not be repeated here.

[0262] The area determination unit 123 is used to determine a standard marking area for marking abnormal objects according to the area review result.

[0263] The area determination unit 123 includes: a first review subunit 1231, a second review subunit 1232, and a third review subunit 1233;

[0264] The first audit subunit 1231 is used to send the regional audit result to the first audit terminal if the regional audit result is a candidate annotation region in at least two candidate region images, so that the first audit terminal determines a first candidate audit region for sending to the second audit terminal based on the candidate annotation region in at least two candidate region images; the first candidate audit region is used to instruct the second audit terminal to return a standard annotation region for annotating abnormal objects;

[0265] The second audit subunit 1232 is used to send the regional audit result to the first audit terminal if the regional audit result is a candidate annotation area included in the target candidate area image, so that the first audit terminal determines a second candidate audit area for sending to the second audit terminal based on the candidate annotation area included in the target candidate area image; the second candidate audit area is used to instruct the second audit terminal to return a standard annotation area for annotating abnormal objects;

[0266] The third review subunit 1233 is configured to send the regional review result to the second review terminal if the regional review result is a merged candidate region, so that the second review terminal returns a standard marking region for marking abnormal objects according to the merged candidate region.

[0267] The specific implementation of the first audit subunit 1231, the second audit subunit 1232 and the third audit subunit 1233 can be found in the above Figure 6 The description of step S1023 in the corresponding embodiment will not be repeated here.

[0268] The specific implementation of the image acquisition unit 121, the area review unit 122 and the area determination unit 123 can be found in the above Figure 6 The description of step S1021 to step S1023 in the corresponding embodiment will not be repeated here.

[0269] The first sending module 13 is used to adjust the model parameters of the initial network model based on the predicted annotation area and the standard annotation area to obtain the first updated model parameters, and send the first updated model parameters to the server so that the server performs parameter fusion on the first updated model parameters and the second updated model parameters to generate target model parameters; the second updated model parameters are provided by the second annotation terminal;

[0270] The first sending module 13 includes: an indicator determination unit 131, a parameter adjustment unit 132;

[0271] An index determination unit 131, configured to determine a regional statistical index associated with the original image based on the predicted annotation region and the standard annotation region;

[0272] The number of original images is at least two, and the at least two original images include the original image T i ; i is a positive integer less than or equal to the number of images of at least two original images; original image T i The number of regions corresponding to the predicted annotation area is N, and the original image T i The number of corresponding standard marked areas is M; N is a positive integer, and M is a positive integer;

[0273] The index determination unit 131 includes: a first fusion subunit 1311, a second fusion subunit 1312, a coincidence determination subunit 1313, and an index determination subunit 1314;

[0274] The first fusion subunit 1311 is used to transform the original image T i The corresponding N predicted and labeled regions are regionally fused to obtain the predicted fused regions corresponding to the N predicted and labeled regions;

[0275] The second fusion subunit 1312 is used to transform the original image T i The corresponding M standard annotated areas are regionally fused to obtain the standard fused areas corresponding to the M standard annotated areas;

[0276] The overlap determination subunit 1313 is used to determine the overlap between the predicted fusion region and the standard fusion region as the original image T i The corresponding first region overlap;

[0277] Among them, the image corresponding to the predicted annotation area is the predicted area image, and the image corresponding to the standard annotation area is the standard area image;

[0278] The overlap determination subunit 1313 is specifically used to obtain the position information L of the predicted fusion area in the predicted area image. 1 , obtain the position information L of the standard fusion area in the standard area image 2 ;

[0279] The overlap determination subunit 1313 is further specifically configured to determine the overlap according to the position information L 1 and location information L 2 , determine the overlap position information L between the predicted fusion area and the standard fusion area 3 ;

[0280] The overlap determination subunit 1313 is further specifically configured to determine the overlap according to the position information L 1 , location information2 and coincidence position information L 3 , determine the regional overlap between the predicted fusion region and the standard fusion region;

[0281] The overlap determination subunit 1313 is further configured to determine the region overlap as the original image T i The corresponding first area overlap.

[0282] The index determination subunit 1314 is used to determine the regional statistical index associated with at least two original images according to the overlap degree of the first region corresponding to each original image.

[0283] The index determination subunit 1314 is specifically used to average the first region overlaps corresponding to each original image to obtain average region overlaps corresponding to at least two original images;

[0284] The indicator determination subunit 1314 is further specifically configured to determine the average region overlap as a region statistical indicator associated with at least two original images.

[0285] The indicator determination subunit 1314 is specifically used to obtain the first region coincidence greater than the first coincidence threshold value from the first region coincidences corresponding to each original image, and count the number of first region coincidences greater than the first coincidence threshold value;

[0286] The index determination subunit 1314 is further specifically configured to determine the index coincidence ratio corresponding to the at least two original images based on the coincidence number and the number of images of the at least two original images;

[0287] The indicator determination subunit 1314 is further specifically configured to determine the indicator coincidence ratio as a regional statistical indicator associated with at least two original images.

[0288] The specific implementation of the first fusion subunit 1311, the second fusion subunit 1312, the coincidence determination subunit 1313 and the index determination subunit 1314 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0289] The parameter adjustment unit 132 is used to adjust the model parameters of the initial network model if the regional statistical indicators meet the model update conditions, obtain first updated model parameters, and send the first updated model parameters to the server.

[0290] The parameter adjustment unit 132 includes: a position acquisition subunit 1321, a loss value determination subunit 1322, a parameter adjustment subunit 1323, and a parameter sending subunit 1324;

[0291] The position acquisition subunit 1321 is used to acquire the standard region position information of the standard marked region and the predicted region position information of the predicted marked region if the regional statistical index meets the model update condition;

[0292] The loss value determination subunit 1322 is used to determine the regional loss value of the initial network model according to the standard regional location information and the predicted regional location information;

[0293] The parameter adjustment subunit 1323 is used to adjust the model parameters in the initial network model according to the regional loss value, and when the adjusted initial network model meets the model convergence condition, the adjusted initial network model is determined as the updated network model;

[0294] The parameter sending subunit 1324 is used to obtain a first updated model parameter for updating the network model, and send the first updated model parameter to the server.

[0295] The specific implementation of the position acquisition subunit 1321, the loss value determination subunit 1322, the parameter adjustment subunit 1323 and the parameter sending subunit 1324 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0296] The first receiving module 14 is used to receive the target model parameters returned by the server, and update the initial network model according to the target model parameters to obtain the target network model for marking the region of the abnormal object.

[0297] The specific implementation of the model acquisition module 11, the regional audit module 12, the first sending module 13 and the first receiving module 14 can be found in the above Figure 3 In the corresponding embodiment, steps S101 to S104 and the above Figure 6 The description of step S1021 to step S1023 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0298] For further information, see Fig.12 , Fig.12 2 is a schematic diagram of the structure of an image data processing device provided in an embodiment of the present application. The image data processing device 2 may include: a model sending module 21, a second receiving module 22, and a second sending module 23;

[0299] The model sending module 21 is used for the server to send the initial network model to the first marking terminal and the second marking terminal, so that the first marking terminal performs region marking on the original image based on the initial network model to obtain a predicted marking region for marking abnormal objects in the original image;

[0300] The second receiving module 22 is used to receive the first updated model parameter returned by the first annotation terminal and the second updated model parameter returned by the second annotation terminal; the first updated model parameter is obtained by adjusting the model parameter of the initial network model by the first annotation terminal based on the predicted annotation area and the standard annotation area for annotating the abnormal object; the standard annotation area is obtained by performing regional review on the candidate annotation area after obtaining the candidate annotation area provided by the annotation object for annotating the abnormal object in the original image;

[0301] The second sending module 23 is used to perform parameter fusion on the first updated model parameters and the second updated model parameters to generate target model parameters, and send the target model parameters to the first labeling terminal and the second labeling terminal; the target model parameters are used to instruct the first labeling terminal to update the initial network model through the target model parameters to obtain a target network model for regional labeling of abnormal objects.

[0302] The specific implementation of the model sending module 21, the second receiving module 22 and the second sending module 23 can be found in the above Fig. 9 The description of steps S201 to S208 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0303] For further information, see Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0304] In such Fig.13 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005.

[0305] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 , Figure 6 or Fig. 9 The description of the image data processing method in the corresponding embodiment can also be performed as described above. Fig.11 In the corresponding embodiment, the image data processing device 1 or Fig.12 The description of the image data processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0306] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the image data processing device 1 or the image data processing device 2 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 , Figure 6 or Fig. 9 The description of the image data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0307] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 , Figure 6 or Fig. 9 The description of the image data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0308] For further information, see Fig.14 , Fig.14300 is a schematic diagram of the structure of an image data processing system provided in an embodiment of the present application. The image data processing system 300 may include an image data processing device 300a and an image data processing device 300b. The image data processing device 300a may be the above-mentioned Fig.11 In the corresponding embodiment of the image data processing device 1, it can be understood that the image data processing device 300a can be integrated in the above Figure 2 The corresponding user terminal 20b in the embodiment will not be described in detail here. Fig.12 In the corresponding embodiment of the image data processing device 2, it can be understood that the image data processing device 300b can be integrated in the above Figure 2 The server 20a in the corresponding embodiment will not be described in detail here. In addition, the description of the beneficial effects of the same method will not be described in detail. For technical details not disclosed in the image data processing system embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0309] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0310] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for processing image data, characterized in that: include: The first annotation terminal obtains an initial network model from a server, and annotates regions of the original image based on the initial network model to obtain a predicted annotation region for annotating abnormal objects in the original image; Obtain a candidate annotation region provided by the annotation object for annotating the abnormal object in the original image, perform a region review on the candidate annotation region, and obtain a standard annotation region for annotating the abnormal object; the number of the original images is at least two, and the at least two original images include the original image T i ; i is a positive integer less than or equal to the number of images of the at least two original images; the original image T i The number of regions corresponding to the predicted annotation regions is N, and the original image T i The number of corresponding standard marked areas is M; N is a positive integer, and M is a positive integer; For the original image T i The corresponding N predicted and annotated regions are subjected to regional fusion to obtain the predicted fusion region corresponding to the N predicted and annotated regions, and the original image T is i The corresponding M standard annotated areas are subjected to regional fusion to obtain the standard fusion areas corresponding to the M standard annotated areas; The area overlap between the predicted fusion area and the standard fusion area is determined as the original image T i The corresponding first region overlap; Obtaining a first area coincidence greater than a first coincidence threshold value from the first area coincidences corresponding to each original image, and counting the number of first area coincidences greater than the first coincidence threshold value; Determining a coincidence ratio of indicators corresponding to the at least two original images based on the coincidence number and the number of images of the at least two original images; Determining the indicator overlap ratio as a regional statistical indicator associated with the at least two original images; If the regional statistical index is less than the statistical index threshold, it is determined that the regional statistical index meets the model update condition, the model parameters of the initial network model are adjusted according to the regional loss value to obtain the first updated model parameters, and the first updated model parameters are sent to the server so that the server performs parameter fusion on the first updated model parameters and the second updated model parameters to generate target model parameters; the second updated model parameters are provided by the second annotation terminal; the regional loss value is determined according to the standard area position information of the standard annotation area and the predicted area position information of the predicted annotation area; The target model parameters returned by the server are received, and the initial network model is updated according to the target model parameters to obtain a target network model for marking the region of the abnormal object.

2. The method according to claim 1, characterized in that The image corresponding to the predicted marked area is the predicted area image, and the image corresponding to the standard marked area is the standard area image; The region overlap between the predicted fusion region and the standard fusion region is determined as the original image T i The corresponding first area overlap includes: Acquire position information L1 of the predicted fusion area in the predicted area image, and acquire position information L2 of the standard fusion area in the standard area image; Determine, according to the position information L1 and the position information L2, the overlapping position information L3 between the predicted fusion area and the standard fusion area; Determining a degree of regional overlap between the predicted fusion region and the standard fusion region according to the position information L1, the position information L2 and the overlap position information L3; The region overlap is determined as the original image T i The corresponding first area overlap.

3. The method according to claim 1, characterized in that If the regional statistical index satisfies the model update condition, adjusting the model parameters of the initial network model according to the regional loss value to obtain first updated model parameters, and sending the first updated model parameters to the server, including: If the regional statistical index meets the model update condition, obtaining the standard region position information of the standard marked region and the predicted region position information of the predicted marked region; Determining a regional loss value of the initial network model according to the standard regional location information and the predicted regional location information; Adjusting the model parameters in the initial network model according to the regional loss value, and determining the adjusted initial network model as the updated network model when the adjusted initial network model meets the model convergence condition; A first updated model parameter of the updated network model is obtained, and the first updated model parameter is sent to the server.

4. The method according to any one of claims 1 to 3, characterized in that: The first annotation terminal obtains an initial network model from a server, and annotates a region of the original image based on the initial network model to obtain a predicted annotation region for annotating an abnormal object in the original image, including: The first annotation terminal obtains an initial network model from a server, inputs an original image into the initial network model, and extracts an image feature vector of the original image through the initial network model; Determining, based on the image feature vector, predicted region position information associated with the abnormal object in the original image; The original image is region-annotated according to the predicted region position information to generate a predicted annotated region for annotating the abnormal object.

5. The method according to any one of claims 1 to 3, characterized in that: The number of the marked objects is at least two; The step of obtaining a candidate annotation region provided by the annotation object for annotating the abnormal object in the original image, and performing a region review on the candidate annotation region to obtain a standard annotation region for annotating the abnormal object includes: Acquire at least two candidate region images provided by at least two annotation objects; each of the at least two candidate region images includes a candidate annotation region for annotating the abnormal object in the original image; Determine a region review result according to the candidate annotated regions included in each candidate region image and the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image; According to the area audit result, a standard marking area for marking the abnormal object is determined.

6. The method according to claim 5, characterized in that The determining of the region review result according to the candidate annotated regions included in each candidate region image and the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image includes: Determine the number of candidate regions corresponding to the candidate annotated regions included in each candidate region image, and compare the numbers of at least two candidate regions to obtain a number comparison result; If the number comparison result indicates that there are at least S identical candidate area numbers in the at least two candidate area numbers, the at least S identical candidate area numbers are determined as the target candidate area number, and the area audit result is determined based on the candidate area images associated with the target candidate area number; S is a positive integer less than or equal to the number of images of the at least two candidate area images; If the quantity comparison result indicates that the at least S identical candidate region quantities do not exist in the at least two candidate region quantities, the candidate annotated regions in the at least two candidate region images are determined as the region review results.

7. The method according to claim 6, characterized in that The determining of the region audit result according to the candidate region images associated with the number of target candidate regions includes: Acquire candidate annotated regions included in every two candidate region images in the target candidate region image, and determine a second region overlap between the candidate annotated regions included in every two candidate region images; the target candidate region image is a candidate region image associated with the number of target candidate regions; Compare the second region coincidence degree with the second coincidence degree threshold value respectively to obtain a coincidence degree comparison result; If the overlap comparison result indicates that in the target candidate area image, there are at least P candidate area images whose second area overlap corresponding to every two candidate area images is greater than the second overlap threshold, then the candidate annotation areas included in the at least P candidate area images are merged to obtain a merged candidate area, and the merged candidate area is determined as the area review result; P is a positive integer less than or equal to the number of images of the target candidate area image; If the overlap comparison result indicates that in the target candidate area image, there is no second area overlap corresponding to every two candidate area images among the at least P candidate area images that is greater than the second overlap threshold, then the candidate annotation area included in the target candidate area image is determined as the area review result.

8. The method according to claim 7, characterized in that Determining a standard marking area for marking the abnormal object according to the area review result includes: If the regional audit result is a candidate annotated area in the at least two candidate area images, the regional audit result is sent to the first audit terminal, so that the first audit terminal determines a first candidate audit area for sending to the second audit terminal based on the candidate annotated areas in the at least two candidate area images; the first candidate audit area is used to instruct the second audit terminal to return a standard annotated area for annotating the abnormal object; If the area audit result is a candidate annotation area included in the target candidate area image, the area audit result is sent to the first audit terminal, so that the first audit terminal determines a second candidate audit area for sending to the second audit terminal based on the candidate annotation area included in the target candidate area image; the second candidate audit area is used to instruct the second audit terminal to return a standard annotation area for annotating the abnormal object; If the region audit result is the merged candidate region, the region audit result is sent to the second audit terminal, so that the second audit terminal returns a standard marking region for marking the abnormal object according to the merged candidate region.

9. A method for processing image data, characterized in that: include: The server sends the initial network model to the first annotation terminal and the second annotation terminal, so that the first annotation terminal performs region annotation on the original image based on the initial network model to obtain a predicted annotation region for annotating abnormal objects in the original image; Receive a first updated model parameter returned by the first annotation terminal and a second updated model parameter returned by the second annotation terminal; the first updated model parameter is obtained by adjusting the model parameter of the initial network model by the first annotation terminal based on the regional loss value when the regional statistical index meets the model update condition, and the regional loss value is determined according to the standard region position information of the standard annotation region used to annotate the abnormal object and the predicted region position information of the predicted annotation region; the standard annotation region is obtained by performing regional review on the candidate annotation region after obtaining the candidate annotation region provided by the annotation object for annotating the abnormal object in the original image; the regional statistical index that meets the model update condition is less than the statistical index threshold; the number of the original images is at least two, and the at least two original images include the original image T i ; i is a positive integer less than or equal to the number of images of the at least two original images; the original image T i The number of regions corresponding to the predicted annotation regions is N, and the original image T i The number of regions of the corresponding standard annotated region is M; N is a positive integer, and M is a positive integer; the regional statistical index refers to the index coincidence ratio determined according to the number of coincidences and the number of images of the at least two original images, and the number of coincidences refers to the number of first region coincidences greater than the first coincidence threshold value in the first region coincidences corresponding to each original image; wherein the original image T i The corresponding first region coincidence refers to the region coincidence between the predicted fusion region and the standard fusion region, wherein the predicted fusion region is obtained by comparing the original image T i The corresponding N predicted annotation regions are obtained by regional fusion, and the standard fusion region is obtained by merging the original image T i The corresponding M standard annotated regions are obtained by regional fusion; The first update model parameters and the second update model parameters are fused to generate target model parameters, and the target model parameters are sent to the first annotation terminal and the second annotation terminal; the target model parameters are used to instruct the first annotation terminal to update the initial network model through the target model parameters to obtain a target network model for regional annotation of the abnormal object.

10. A computer device, characterized in that: include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 9.

Images