Image labeling method and device, computer device, and storage medium

Abstract

Embodiments of the present application disclose a kind of image labeling method, device, computer equipment and storage medium, belong to computer technical field.The method comprises: obtaining the first sample image of source domain, the first sample image of first sample image and the multiple second sample images of target domain, select target sample image from multiple second sample images, obtain the second sample image of target sample image, call image labeling model, respectively label first sample image and target sample image, obtain the first predicted labeling image of first sample image and the second predicted labeling image of target sample image, based on the difference between first labeling image and first predicted labeling image, the difference between second labeling image and second predicted labeling image, image labeling model is trained, from target domain selects and source domain most dissimilar image to train image labeling model, improve the model performance of image labeling model on target domain.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to an image labeling method and device, computer equipment and storage medium. BACKGROUND

[0002] Image semantic segmentation is used to segment an image into regions with different semantic information, and can be applied to automatic driving, medical image analysis, face recognition and other scenarios. In various scenarios, an image labeling model is usually used to label regions with different semantic information in an image, i.e., to obtain a labeled image of the image. Since images in different fields in the same scenario may differ, to ensure image segmentation effect, an image labeling model trained using source domain images is usually migrated to a target domain, so that the migrated image labeling model can implement an image segmentation task in the target domain. However, in the migration process of the image labeling model, how to improve the model performance of the migrated image labeling model in the target domain is a problem to be solved at present. SUMMARY

[0003] Embodiments of the present application provide an image labeling method and device, computer equipment and storage medium, which can improve the model performance of an image labeling model in a target domain. The technical solution is as follows:

[0004] In one aspect, an image labeling method is provided, which includes:

[0005] obtaining a first sample image of a source domain, a first labeled image of the first sample image, and a plurality of second sample images of a target domain;

[0006] selecting a target sample image from the plurality of second sample images, obtaining a second labeled image of the target sample image, the target sample image being a second sample image in the plurality of second sample images that has the smallest similarity with the first sample image;

[0007] calling an image labeling model to label the first sample image and the target sample image respectively, to obtain a first predicted labeled image of the first sample image and a second predicted labeled image of the target sample image;

[0008] training the image labeling model based on differences between the first labeled image and the first predicted labeled image, and differences between the second labeled image and the second predicted labeled image, the image labeling model being used to label images of the target domain.

[0009] In a possible implementation, after the image labeling model is trained based on the difference between the first labeled image and the first predicted labeled image and the difference between the second labeled image and the second predicted labeled image, the method further includes:

[0010] The trained image labeling model is called to label a target image in the target domain, and a labeled image of the target image is obtained.

[0011] In another aspect, an image labeling apparatus is provided, and the apparatus includes:

[0012] An acquisition module is configured to acquire a first sample image in a source domain, a first labeled image of the first sample image, and a plurality of second sample images in a target domain.

[0013] The acquisition module is further configured to select a target sample image from the plurality of second sample images, and acquire a second labeled image of the target sample image, the target sample image being a second sample image in the plurality of second sample images that has a minimum similarity with the first sample image.

[0014] A labeling module is configured to call an image labeling model, and label the first sample image and the target sample image respectively, to obtain a first predicted labeled image of the first sample image and a second predicted labeled image of the target sample image.

[0015] A training module is configured to train the image labeling model based on a difference between the first labeled image and the first predicted labeled image and a difference between the second labeled image and the second predicted labeled image, the image labeling model being used to label images in the target domain.

[0016] In a possible implementation, the apparatus further includes:

[0017] The acquisition module is further configured to acquire first image features of the plurality of first sample images and second image features of the plurality of second sample images.

[0018] A clustering module is configured to cluster the plurality of first sample images based on the acquired first image features, to obtain at least one first clustering center.

[0019] A determination module is configured to determine, for any second sample image, a distance between a second image feature of the second sample image and each first clustering center, and determine a similarity between the second sample image and the first sample image based on a minimum distance in the determined distances, the minimum distance corresponding to the second sample image being in a negative correlation with the similarity corresponding to the second sample image.

[0020] In another possible implementation manner, the obtaining module comprises:

[0021] The feature extraction unit is configured to invoke a feature extraction sub-model in the image annotation model, and perform feature extraction on each first sample image to obtain third image features of each first sample image.

[0022] The fusion unit is configured to fuse the third image features corresponding to each first sample image with the corresponding first annotation image to obtain first image features of each first sample image.

[0023] In another possible implementation manner, the first annotation image comprises at least one region corresponding to a category; and the fusion unit is configured to, for any first sample image, extract a sub-annotation image corresponding to the at least one category from the first annotation image of the first sample image, the sub-annotation image being used to indicate pixel points belonging to the corresponding category; fuse each sub-annotation image corresponding to a category with the third image features of the first sample image to obtain fourth image features corresponding to each category; and splice the fourth image features corresponding to the at least one category to obtain the first image features of the first sample image.

[0024] In another possible implementation manner, the fusion unit is configured to, for any sub-annotation image corresponding to a category, multiply a pixel value of each pixel point in the sub-annotation image with a corresponding feature value to obtain a product corresponding to the each pixel point, the feature value corresponding to any pixel point being a feature value in the third image features located at the same position as the pixel point; determine a ratio between the product corresponding to the each pixel point and a number of pixel points, the number of pixel points being a number of pixel points belonging to the corresponding category in the sub-annotation image; and construct the fourth image features corresponding to the category by using the ratio corresponding to the each pixel point.

[0025] In another possible implementation manner, the obtaining module comprises:

[0026] The feature extraction unit is configured to invoke a feature extraction sub-model in the image annotation model, and perform feature extraction on each second sample image to obtain fifth image features of each second sample image.

[0027] The first obtaining unit is configured to obtain a soft annotation image of each second sample image, the soft annotation image being obtained by annotating a second sample image by using the image annotation model before training the image annotation model.

[0028] The fusion unit is configured to fuse the fifth image features of each second sample image with the corresponding soft annotation image to obtain second image features of each second sample image.

[0029] In another possible implementation manner, the training module comprises:

[0030] The second obtaining unit is configured to, in the process of training the image labeling model, obtain at least one second clustering center corresponding to the plurality of second sample images in the current iteration, the at least one second clustering center being obtained based on clustering of second image features of the second sample images;

[0031] The second obtaining unit is further configured to obtain distances between the second image features of each second sample image in the current iteration and the at least one second clustering center.

[0032] The training unit is configured to train the image labeling model in the current iteration based on the difference between the first labeled image and the first predicted labeled image, the difference between the second labeled image and the second predicted labeled image, and the distances corresponding to the plurality of second sample images.

[0033] In another possible implementation manner, the second obtaining unit is configured to, if the current iteration is a first iteration in the training process, obtain the second image features of the plurality of second sample images in the current iteration; and cluster the plurality of second sample images based on the obtained plurality of second image features to obtain the at least one second clustering center.

[0034] In another possible implementation manner, the second obtaining unit is further configured to, if the current iteration is not the first iteration in the training process, assign each second image feature to a nearest second clustering center based on distances between each second image feature and each second clustering center in a previous iteration; update each second clustering center based on the second image features corresponding to the second clustering center; and determine the updated second clustering center as the second clustering center corresponding to the current iteration.

[0035] In another possible implementation manner, the obtaining module is further configured to obtain a soft labeled image of a third sample image, the third sample image being a second sample image other than the target sample image in the plurality of second sample images, the soft labeled image being obtained by labeling the third sample image by the image labeling model before training the image labeling model.

[0036] The training unit is configured to invoke the image labeling model to label the third sample image to obtain a third predicted labeling image of the third sample image; and train the image labeling model of the current iteration based on a difference between the first labeling image and the first predicted labeling image, a difference between the second labeling image and the second predicted labeling image, the distance corresponding to each second sample image, and a difference between the soft labeling image and the third predicted labeling image.

[0037] In another possible implementation, the training module is further configured to perform adversarial training on the image labeling model and a discriminator model based on the first sample image, the first labeling image, and the plurality of second sample images, the discriminator model being configured to determine whether a labeling image output by the image labeling model is a labeling image of the first sample image.

[0038] In another possible implementation, the training module is configured to invoke the image labeling model to label the first sample image and the plurality of second sample images respectively to obtain a fourth predicted labeling image of the first sample image and a fifth predicted labeling image of each second sample image; invoke the discriminator model to determine the fourth predicted labeling image and the fifth predicted labeling image to obtain a determination result; and train the image labeling model and the discriminator model based on a difference between the fourth predicted labeling image and the first labeling image and the determination result.

[0039] In another possible implementation, the labeling module is further configured to invoke the trained image labeling model to label a target image of a target domain to obtain a labeling image of the target image.

[0040] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the image labeling method according to the above aspects.

[0041] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, which is loaded and executed by a processor to implement the operations performed in the image labeling method according to the above aspects.

[0042] In still another aspect, a computer program product or computer program is provided, which includes computer program code stored in a computer readable storage medium. A processor of a computer device reads the computer program code from the computer readable storage medium, and the processor executes the computer program code to cause the computer device to implement operations performed in the image labeling method as described in the above aspects.

[0043] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0044] The method and device, computer device and storage medium provided by the embodiments of the present application, because of the differences between the images of the source domain and the target domain, the images of the target domain have unique information of the target domain, therefore, the sample image most dissimilar to the sample images of the source domain is selected from the target domain, and the selected sample image and the sample images and labeled images of the source domain are used to train the image labeling model, so that the image labeling model can learn the unique information of the sample images of the target domain, thereby improving the applicability of the image labeling model in the target domain, improving the model performance of the image labeling model in the target domain, and improving the labeling accuracy of the image labeling model in the target domain. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 is a structural schematic diagram of a data sharing system provided by the embodiments of the present application;

[0047] Figure 2 is a structural schematic diagram of a blockchain provided by the embodiments of the present application;

[0048] Figure 3 is a flowchart of a new block generation provided by the embodiments of the present application;

[0049] Figure 4 is a structural schematic diagram of an implementation environment provided by the embodiments of the present application;

[0050] Figure 5 is a flowchart of an image labeling method provided by the embodiments of the present application;

[0051] Figure 6 is a flowchart of an image labeling method provided by the embodiments of the present application;

[0052] Figure 7 is a flowchart of acquiring a target sample image provided by an embodiment of the present application;

[0053] Figure 8 is a flowchart of acquiring a soft-labeled image and a second clustering center provided by an embodiment of the present application;

[0054] Figure 9 is a flowchart of training an image labeling model provided by an embodiment of the present application;

[0055] Figure 10 is a schematic diagram of a plurality of labeled images provided by an embodiment of the present application;

[0056] Figure 11 is a structural schematic diagram of an image labeling apparatus provided by an embodiment of the present application;

[0057] Figure 12 is a structural schematic diagram of an image labeling apparatus provided by an embodiment of the present application;

[0058] Figure 13 is a structural schematic diagram of a terminal provided by an embodiment of the present application;

[0059] Figure 14 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions and advantages of embodiments of the present application clearer, the following will further describe the embodiments of the present application in conjunction with the accompanying drawings.

[0061] The terms "first", "second", "third", "fourth", "fifth" and the like as used herein can be used to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the present application, a first image feature can be referred to as a second image feature, and similarly, a second image feature can be referred to as a first image feature.

[0062] The terms "at least one", "multiple", "each", "any" used in the present application include one, two or more than two, multiple includes two or more than two, and each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, a plurality of second sample images includes 3 second sample images, and each refers to each of the 3 second sample images, any refers to any one of the 3 second sample images, which can be the first second sample image, or the second, or the third second sample image.

[0063] Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine which can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0064] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation and other major directions.

[0065] Computer vision (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation and other technologies, as well as common face recognition, fingerprint recognition and other biometric identification technologies.

[0066] Machine learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0067] The automatic driving technology usually includes high-precision map, environment perception, behavior decision, path planning, motion control and other technologies, and has a wide application prospect.

[0068] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, Internet of vehicles, automatic driving, intelligent transportation and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.

[0069] The scheme provided by the embodiment of the present application can train an image labeling model based on computer vision technology and machine learning technology of artificial intelligence, and can realize the labeling task of the image by using the trained image labeling model, and can be applied in various scenes subsequently.

[0070] The image labeling method provided by the embodiment of the present application can be executed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto.

[0071] In some embodiments, the computer program related to the embodiment of the present application can be deployed to execute on one computer device, or to execute on multiple computer devices located in one place, or to execute on multiple computer devices distributed in multiple places and interconnected through a communication network. The multiple computer devices distributed in multiple places and interconnected through a communication network can constitute a blockchain system.

[0072] Referring to Figure 1The data sharing system shown in the figure is a system for data sharing between nodes, which can include a plurality of nodes 101, which can be referred to as clients in the data sharing system. Each node 101 can receive input information during normal operation and maintain shared data in the data sharing system based on the received input information. To ensure information exchange in the data sharing system, each node in the data sharing system can have an information connection between nodes, and information can be transmitted between nodes through the information connection. For example, when any node in the data sharing system receives input information, other nodes in the data sharing system obtain the input information according to a consensus algorithm, and store the input information as data in the shared data, so that the data stored on all nodes in the data sharing system is consistent.

[0073] Each node in the data sharing system has a corresponding node identifier, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system, so that the generated block can be broadcast to other nodes in the data sharing system according to the node identifiers of other nodes. Each node can maintain a node identifier list as shown in the following table, and store the node name and node identifier in the node identifier list. The node identifier can be an IP (Internet Protocol) address or any other information that can be used to identify the node. In Table 1, only the IP address is used as an example for illustration.

[0074] Table 1

[0075] Node name Node identification Node 1 117.114.151.174 Node 2 117.116.189.145 … … Node N 119.123.789.258

[0076] Each node in the data sharing system stores a same block chain. The block chain is composed of a plurality of blocks, as shown in Figure 2 The block chain is composed of a plurality of blocks, and the genesis block includes a block header and a block body. The block header stores an input information feature value, a version number, a timestamp, and a difficulty value, and the block body stores input information. The next block of the genesis block takes the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information feature value of the current block, the block header feature value of the parent block, the version number, the timestamp, and the difficulty value, and so on. Thus, the block data stored in each block in the block chain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0077] When generating each block in the block chain, refer to Figure 3The node where the blockchain is located checks the input information when receiving the input information, stores the input information into the memory pool after completing the checking, and updates the hash tree used for recording the input information; then, the update timestamp is updated as the time when the input information is received, and different random numbers are tried to perform feature value calculation multiple times, so that the calculated feature value can satisfy the following formula:

[0078] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))<TARGET

[0079] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))<TARGET

[0080] In this way, when the random number satisfying the above formula is calculated, the information can be stored correspondingly to generate the block header and the block body, and the current block is obtained. Subsequently, the node where the blockchain is located sends the newly generated block to other nodes in the data sharing system according to the node identifiers of the other nodes in the data sharing system, and the other nodes check the newly generated block and add the newly generated block to the stored blockchain after completing the checking.

[0081] Figure 4 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Referring to Figure 4 The implementation environment includes a terminal 401 and a server 402. The terminal 401 and the server 402 are connected through a wireless or wired network. Optionally, the terminal 401 is installed with a target application provided by the server 402, and the terminal 401 can realize functions such as data transmission and message interaction through the target application. Optionally, the target application is a target application in the operating system of the terminal 401, or a target application provided by a third party. For example, the target application has the function of image labeling, that is, the target application can label the regions with different meanings in an image. Of course, the target application can also have other functions, such as a comment function, a shopping function, a navigation function, a game function, etc.

[0082] In the embodiment of the present application, the terminal 401 server 402 provides sample images of the source domain or sample images of the target domain, and the server 402 trains the image labeling model based on the obtained images. After training the image labeling model, the terminal 401 can complete the image labeling task by using the trained image labeling model. For example, the server 402 shares the trained image labeling model with the terminal 401, and the terminal 401 can deploy the trained image labeling model to complete the image labeling task based on the trained image labeling model; or the terminal 401 sends the image to be labeled to the server 402 through interaction with the server 402, and the server 402 labels the image sent by the terminal 401 based on the trained image labeling model, and returns the labeled image to the terminal 401.

[0083] The method provided by the embodiment of the present application can be used in various scenarios.

[0084] For example, in the automatic driving scenario:

[0085] Since the scenes in different cities may be different, the source domain includes street view images and labeled images in city 1, and the current image labeling model is suitable for city 1, and the target domain includes street view images in city 2 to be migrated. By using the image labeling method provided by the embodiment of the present application, the image labeling model suitable for the target domain is trained based on the sample images of the source domain, the labeled images and the sample images of the target domain, and then the trained image labeling model is called to complete the labeling task of the street view images in city 2, so that the labeled images of the street view images in city 2 are used to identify safe street areas in the future, so as to ensure that the car can drive safely and automatically.

[0086] For another example, in the medical scenario:

[0087] Since the medical images in different hospitals may be different, the medical images of the target hospital are used as sample images of the target domain, and the medical images and labeled images of other hospitals that have been trained are used as the source domain. By using the image labeling method provided by the embodiment of the present application, the image labeling model suitable for the target hospital can be trained, and then the trained image labeling model can be called to complete the labeling task of the medical images in the target hospital, so that the labeled images of the medical images in the target hospital are used to identify different parts and segment the parts that need to be analyzed from the medical images in the future.

[0088] Figure 5 is a flowchart of an image labeling method provided by the embodiment of the present application, executed by a computer device, as shown in Figure 5 The method comprises the following steps.

[0089] 501、The computer device obtains a first sample image of a source domain, a first label image of the first sample image, and a plurality of second sample images of a target domain.

[0090] In the embodiments of the present application, the source domain and the target domain belong to different domains under the same scene, that is, the images contained in the source domain and the target domain have similarities, but the images in the source domain and the target domain are not completely the same. For example, in the automatic driving scene, the images in the source domain and the target domain are both street view images, the street view images in the source domain may be street view images in city 1, and the street view images in the target domain may be street view images in city 2; or, the street view images in the source domain are virtual street view images, and the street view images in the target domain are real street view images. For another example, in the medical scene, the images in the source domain and the target domain are both medical images, the medical images in the source domain are medical images belonging to hospital 1, and the medical images in the target domain are medical images belonging to hospital 2.

[0091] The first label image of the first sample image is used to indicate the categories to which the respective pixel points in the first sample image belong, the categories are used to distinguish the objects contained in the first sample image, the pixel points belonging to different categories are used to describe different objects, for example, the first sample image is a street view image, the street view image may contain streets, bicycles, buildings, etc., that is, the streets, bicycles, and buildings in the street view image all belong to different categories, and therefore, in the first sample image, the categories to which the pixel points describing the streets belong are different from the categories to which the pixel points describing the bicycles belong. The first sample image of the source domain and the second sample image of the target domain belong to the same image type, for example, the first sample image and the second sample image both belong to the street view image type, or both belong to the medical image type, etc. The first sample image is an image contained in the source domain, and the second sample image is an image contained in the target domain.

[0092] 502、The computer device selects a target sample image from the plurality of second sample images, and obtains a second label image of the target sample image, the target sample image being a second sample image having the smallest similarity to the first sample image among the plurality of second sample images.

[0093] The second label image is used to indicate the categories to which the respective pixel points in the target sample image belong.

[0094] In the embodiments of the present application, since the images in the target domain have unique information of the target domain, by selecting the target sample image having the smallest similarity to the first sample image of the source domain from the plurality of second sample images, it is ensured that the image labeling model trained based on the target sample image can learn the unique information of the target domain, so as to improve the applicability of the trained image labeling model in the target domain, thereby improving the model performance of the image labeling model in the target domain.

[0095] 503、The computer device invokes the image labeling model to label the first sample image and the target sample image respectively, to obtain a first predicted labeling image of the first sample image and a second predicted labeling image of the target sample image.

[0096] The image labeling model is used for labeling images, and the first predicted labeling image and the second predicted labeling image are both obtained by the image labeling model. The first predicted labeling image is used to indicate the category to which each pixel point in the first sample image is predicted by the model, and the second predicted labeling image is used to indicate the category to which each pixel point in the target sample image is predicted by the model.

[0097] 504、The computer device trains the image labeling model based on the difference between the first labeling image and the first predicted labeling image, and the difference between the second labeling image and the second predicted labeling image. The image labeling model is used for labeling images in the target domain.

[0098] Since the difference between the first labeling image and the first predicted labeling image, and the difference between the second labeling image and the second predicted labeling image can reflect the accuracy of the image labeling model, training the image labeling model based on the difference between the first labeling image and the first predicted labeling image, and the difference between the second labeling image and the second predicted labeling image can improve the accuracy of the image labeling model.

[0099] The method provided by the embodiments of the present application can select the sample image that is least similar to the sample image in the source domain from the target domain, and train the image labeling model by using the selected sample image and the sample image and the labeling image in the source domain, so that the image labeling model can learn the unique information of the sample image in the target domain, thereby improving the applicability of the image labeling model in the target domain, improving the model performance of the image labeling model in the target domain, and improving the labeling accuracy of the image labeling model in the target domain.

[0100] Based on the embodiments shown in Figure 5 On the basis of the embodiments shown in

[0101] Figure 6 is a flowchart of an image labeling method provided by the embodiments of the present application, executed by a computer device, as shown in Figure 6 The method comprises the following steps.

[0102] 601、The computer device obtains a plurality of first sample images of a source domain, a first label image of each first sample image, and a plurality of second sample images of a target domain.

[0103] Optionally, the first label image is obtained by manual labeling, for example, the first label image is obtained by labeling the first sample image by an expert.

[0104] Optionally, the first label image includes a region corresponding to at least one category, and a plurality of pixel points located in the same region belong to the same category, which is the category corresponding to the region. In the embodiment of the present application, the category of each pixel point in the first sample image is the same as the category of the pixel point located at the same position in the corresponding first label image, that is, the first label image can indicate the category of each pixel point in the corresponding first sample image. For example, the first sample image is a street view image, and the categories corresponding to the street view image can include street, sidewalk, building, wall, fence, pedestrian, vehicle, etc., and the first label image of the first sample image can indicate which category each pixel point belongs to.

[0105] Optionally, different colors are used in the first label image to represent the categories corresponding to the pixel points, that is, the colors in the regions corresponding to different categories are different. For example, the first sample image is a street view image, and the first label image of the first sample image has a red region representing a street and a green region representing a pedestrian.

[0106] Optionally, the first label image is represented in the form of a matrix, for example, the matrix includes a plurality of numerical values, and the plurality of numerical values respectively represent the categories to which the corresponding pixel points belong. For another example, the plurality of numerical values included in the matrix are arranged according to the positions of the corresponding pixel points on the first label image, and the numerical value 1 is used to represent the category 1 and the numerical value 2 is used to represent the category 2, so that the categories of the pixel points in the first label image can be determined based on the plurality of numerical values in the matrix.

[0107] In the embodiment of the present application, the plurality of first sample images of the source domain, the first label images of the plurality of first sample images, and the plurality of second sample images of the target domain can be stored in the form of a blockchain. The computer device can obtain the first sample images, the first label images of the first sample images, and the second sample images from the blockchain.

[0108] It should be noted that the embodiment of the present application is described by taking the plurality of first sample images of the source domain and the plurality of first label images as an example, and in another embodiment, only the first sample image of the source domain, the first label image of the first sample image, and the plurality of second sample images of the target domain can be obtained.

[0109] 602、The computer device performs adversarial training on the image labeling model and the discrimination model based on the plurality of first sample images, the plurality of first labeled images and the plurality of second sample images.

[0110] In the embodiments of the present application, the image labeling model is used for labeling images, for example, the image labeling model is Deeplab (a kind of semantic segmentation model), PSPNet (Pyramid Scene Parseing Network, Pyramid Pooling model). The image labeling model is a model that has been trained using sample images and labeled images of the source domain, that is, the image labeling model is suitable for the source domain, and based on the image labeling model, the image labeling task of the image of the source domain can be performed. The discrimination model is used to discriminate whether the labeled image output by the image labeling model is the labeled image of the first sample image, that is, the discrimination model is used to distinguish the labeled image of the sample image of the source domain and the labeled image of the sample image of the target domain.

[0111] In the adversarial training process, the image labeling model is used to label the first sample image of the source domain and the second sample image of the target domain respectively, and the discrimination model is used to discriminate whether the labeled image output by the image labeling model is the labeled image of the first sample image. In the process of adversarial training of the image labeling model and the discrimination model, the image labeling model is trained to improve the accuracy of the image labeling model, so that the labeled image output by the image labeling model is as difficult as possible for the discrimination model to distinguish, that is, the discrimination model cannot distinguish whether the labeled image output by the image labeling model is the labeled image of the first sample image; the discrimination model is trained to improve the discrimination ability of the discrimination model, so that the discrimination model can distinguish the labeled image output by the image labeling model as much as possible, that is, the discrimination model can distinguish whether the labeled image output by the image labeling model is the labeled image of the first sample image. In this way, the image labeling model and the discrimination model are trained adversarially, which can improve the accuracy of the image labeling model, and finally make the image labeling model and the discrimination model balanced.

[0112] Because there is a difference between the images of the source domain and the images of the target domain, the current image labeling model may not be suitable for the target domain, therefore, the image labeling model is trained in an adversarial manner first, so that the image labeling model after adversarial training can be suitable for the source domain, that is, the trained image labeling model is called to label the first sample image of the source domain and the second sample image of the target domain respectively, so that the obtained labeled images have similarity, the distance between the labeled images corresponding to the target domain and the labeled images corresponding to the source domain is reduced, and the labeled images corresponding to the target domain present an ordered state, realizing the preheating effect of the image labeling model.

[0113] In a possible implementation, the image labeling model and the discrimination model are adversarially trained by using the predicted labeling image output by the image labeling model and the discrimination result output by the discrimination model, that is, the step 602 includes: calling the image labeling model to label the plurality of first sample images and the plurality of second sample images respectively to obtain a fourth predicted labeling image of each first sample image and a fifth predicted labeling image of each second sample image, calling the discrimination model to discriminate the fourth predicted labeling image and the fifth predicted labeling image to obtain a discrimination result, and training the image labeling model and the discrimination model based on the difference between each fourth predicted labeling image and the corresponding first labeling image and the discrimination result.

[0114] The fourth predicted labeling image is obtained by labeling the first sample image by the image labeling model, and is equivalent to a prediction result of the first sample image by the image labeling model. The fifth predicted labeling image is obtained by labeling the second sample image by the image labeling model, and is equivalent to a prediction result of the second sample image by the image labeling model. Optionally, the fourth predicted labeling image or the fifth predicted labeling image is used to indicate the category to which each pixel point in the corresponding sample image predicted by the model belongs.

[0115] The discrimination result is used to indicate whether the predicted labeling image discriminated by the discrimination model is the labeling image of the first sample image, that is, whether the fourth predicted labeling image discriminated by the discrimination model is the labeling image of the first sample image, and is also used to indicate whether the fifth predicted labeling image discriminated by the discrimination model is the labeling image of the first sample image. Based on the difference between the fourth predicted labeling image and the first labeling image, the accuracy of the image labeling model can be determined, and the accuracy of the image labeling model and the accuracy of the discrimination model can be reflected by the discrimination result. Therefore, the image labeling model and the discrimination model are trained by using the difference between the fourth predicted labeling image and the first labeling image and the discrimination result, so as to improve the accuracy of the image labeling model.

[0116] Optionally, when the image labeling model and the discrimination model are trained, the image labeling model and the discrimination model are trained based on the difference between the fourth predicted labeling image and the first labeling image, the difference between the discrimination result and the labeling result.

[0117] The labeling result is used to indicate that the fourth predicted labeling image is the labeling image of the first sample image, and is also used to indicate that the fifth predicted labeling image is not the labeling image of the first sample image, that is, the labeling result is a real result. By determining the difference between the labeling result and the discrimination result, the accuracy of the discrimination model can be determined. The image labeling model and the discrimination model are adversarially trained based on the difference between the fourth predicted labeling image and the first labeling image and the difference between the discrimination result and the labeling result, so as to improve the accuracy of the image labeling model and the accuracy of the discrimination model.

[0118] Optionally, the image labeling model and the discriminator model are adversarially trained using loss values, that is, a first loss value is determined based on a difference between the fourth predicted labeling image and the first labeling image, a second loss value is determined based on a difference between the discrimination result and the labeling result, and the image labeling model and the discriminator model are trained based on a sum of the first loss value and the second loss value. The image labeling model and the discriminator model are adversarially trained using the determined loss values to ensure the accuracy of the trained models.

[0119] In the embodiments of the present application, the image labeling model trained in the above-described adversarial training manner is equivalent to a preheated network, and subsequently, based on the plurality of first sample images of the source domain, the first labeling image of each first sample image, and the plurality of second sample images of the target domain, the trained image labeling model (i.e., the preheated model) can be trained again on the basis of adversarial training to improve the model performance of the image labeling model in the target domain.

[0120] Moreover, on the basis of adversarial training, steps 603-606 are first performed to select a target sample image with the least similarity to the first sample images of the source domain from the target domain, and then the image labeling model after adversarial training is iteratively trained based on the plurality of first sample images of the source domain, the first labeling image of each first sample image, the target sample image of the target domain, the second labeling image of the target sample image, and the third sample image of the target domain other than the target sample image from the plurality of second sample images of the target domain. The present application is only described by taking one iteration as an example, and the one iteration process is shown in steps 607-610.

[0121] 603. The computer device obtains first image features of the plurality of first sample images and second image features of the plurality of second sample images.

[0122] The first image features are used to describe information contained in the first sample images, and the second image features are used to describe information contained in the second sample images. The computer device obtains the first image features of each first sample image of the source domain and the second image features of each second sample image of the target domain, so as to subsequently select a target sample image with the least similarity to the plurality of first sample images from the plurality of second sample images using the obtained image features.

[0123] In a possible implementation manner, the first image features of the first sample images are obtained using the first labeling images of the first sample images, that is, the process of obtaining the first image features of the plurality of first sample images includes the following steps 6031-6032:

[0124] 6031. A feature extraction sub-model in the image labeling model is invoked to perform feature extraction on each first sample image to obtain third image features of each first sample image.

[0125] In the embodiment of the present application, the image labeling model comprises a feature extraction sub-model, and the feature extraction sub-model is configured to extract features of the image. The image features of each first sample image are extracted by the feature extraction sub-model in the image labeling model, so as to ensure the accuracy of the extracted third image features, and other feature extraction models do not need to be configured to perform feature extraction, thereby saving resources.

[0126] 6032, fuse the third image features corresponding to each first sample image with the corresponding first labeled image to obtain first image features of each first sample image.

[0127] Since the first labeled image is used to indicate the category to which each pixel point in the corresponding first sample image belongs, the category to which each pixel point in the corresponding first sample image belongs is integrated into the obtained first image features by fusing the first labeled image with the corresponding third image features, the information contained in the first image features is enriched, and the accuracy of the first image features is improved.

[0128] In a possible implementation manner, the first image features of the first sample image are obtained by using the sub-labeled image of at least one category contained in the first labeled image and by adopting the fusion-then-stitching manner, that is, the step 6032 comprises the following steps 6033-6035:

[0129] 6033, for any first sample image, extract a sub-labeled image corresponding to at least one category from the first labeled image of the first sample image, and the sub-labeled image is used to indicate the pixel points belonging to the corresponding category.

[0130] Each sub-labeled image is used to indicate the pixel points belonging to the corresponding category, that is, to indicate the category to which the pixel points located at the same position as the pixel points in each sub-labeled image in the corresponding first sample image belong. Alternatively, in the sub-labeled image, the pixel value of the pixel point belonging to the category corresponding to the sub-labeled image is a first value, and the pixel value of the pixel point not belonging to the category corresponding to the sub-labeled image is a second value. For example, the first value is 1, and the second value is 0. In any sub-labeled image, the pixel value of the pixel point belonging to the category corresponding to the sub-labeled image is 1, and the pixel value of the pixel point not belonging to the category corresponding to the sub-labeled image is 0.

[0131] In the embodiment of the present application, the first labeled image comprises a region corresponding to at least one category, and the categories to which the pixel points located in the same region belong are the same. By extracting the sub-labeled image corresponding to each category from the first labeled image, the sub-labeled image corresponding to any category can indicate the pixel points located in the region corresponding to the category in the first labeled image.

[0132] 6034、fuse the sub-labeled image corresponding to each category with the third image feature of the first sample image to obtain a fourth image feature corresponding to each category.

[0133] The fourth image feature is used to describe information contained by the pixel points belonging to the corresponding category in the first sample image.

[0134] Alternatively, the sub-labeled image and the third image feature are fused in units of pixel points, that is, the process of obtaining the fourth image feature includes: for the sub-labeled image corresponding to any category, multiplying the pixel value of each pixel point in the sub-labeled image by the corresponding feature value to obtain a product corresponding to each pixel point, the feature value corresponding to any pixel point being the feature value in the third image feature located at the same position as the pixel point; determining the ratio between the product corresponding to each pixel point and the number of pixel points, the ratio corresponding to each pixel point constituting the fourth image feature corresponding to the category, the number of pixel points being the number of pixel points belonging to the corresponding category in the sub-labeled image.

[0135] In the embodiment of the present application, each pixel point in the sub-labeled image corresponds to the first feature value in the third sample image, and the position of the corresponding feature value in the third sample image is the same as the position of the pixel point in the sub-labeled image. By fusing the pixel value and the feature value located at the same position in the third image feature in units of pixel points, it is ensured that the obtained fourth image feature can highlight the features of the pixel points of the corresponding category, thereby ensuring the accuracy of the fourth image feature.

[0136] Alternatively, for any first sample image and any category, the fourth image feature corresponding to the category satisfies the following relationship:

[0137]

[0138] wherein x s is the first sample image of the source domain, c is used to represent any category in the at least one category included by the first labeled image of the first sample image x s , is used to represent the fourth image feature corresponding to the category c, is used to represent the sub-labeled image corresponding to the category, is used to represent the number of pixel points belonging to the category c in the sub-labeled image . is used to represent pixel-by-pixel multiplication, f E (x s ) is used to represent the third image feature of the first sample image x s , and f E (·) is used to represent the feature extraction sub-model in the image labeling model.

[0139] 6035、stitching the fourth image features corresponding to at least one category to obtain the first image feature of the first sample image.

[0140] For any first sample image, after obtaining the fourth image features corresponding to each category, the fourth image features corresponding to at least one category are stitched, so that the features of multiple categories possessed by the first sample image are integrated into the obtained first image feature, thereby enriching the information contained in the first image feature and improving the accuracy of the first image feature.

[0141] In a possible implementation manner, the fourth image feature is a three-dimensional image feature, that is, the fourth image feature includes multiple two-dimensional image features, and then each fourth image feature is flattened, and then the flattened image features are stitched into the first image feature.

[0142] In the process of flattening each fourth image feature, that is, the multiple two-dimensional image features included in the fourth image feature are stitched to obtain the stitched two-dimensional image features, and the stitched two-dimensional image features are the image features after the fourth image feature is flattened. The first image feature is a two-dimensional image feature.

[0143] In a possible implementation manner, the second image features of the second sample images are obtained by using the soft-labeled images of the second sample images, that is, the process of obtaining the second image features of the multiple second sample images includes the following steps 6036-6038:

[0144] 6036、invoke the feature extraction sub-model in the image labeling model to perform feature extraction on each second sample image to obtain the fifth image feature of each second sample image.

[0145] This step is the same as the above step 6031, and will not be described here.

[0146] 6037、obtain the soft-labeled image of each second sample image, wherein the soft-labeled image is obtained by labeling the second sample image by the image labeling model before training the image labeling model.

[0147] The soft-labeled image of any second sample image is used to indicate the category to which each pixel point in the second sample image belongs.

[0148] In a possible implementation manner, the soft-labeled image is obtained by using the image labeling model, that is, the step 6037 includes: invoking the image labeling model to label the second sample image to obtain the labeled image of the second sample image, and determining the obtained labeled image as the soft-labeled image of the second sample image.

[0149] The image annotation model is either an image annotation model applicable to the source domain, or an image annotation model trained adversarially according to step 602 above.

[0150] 6038. Fuse the fifth image feature of each second sample image with the corresponding soft-labeled image to obtain the second image feature of each second sample image.

[0151] In one possible implementation, the second image features of the second sample image are obtained by using sub-labeled images of at least one category contained in the soft-labeled image and employing a method of first fusing and then stitching them together. That is, step 6038 includes: for any second sample image, extracting at least one sub-labeled image corresponding to a category from the soft-labeled image of the second sample image, the sub-labeled image being used to indicate pixels belonging to the corresponding category; fusing the sub-labeled image corresponding to each category with the fifth image features of the second sample image to obtain a sixth image feature corresponding to each category; and stitching together the sixth image features corresponding to at least one category to obtain the second image features of the second sample image.

[0152] Optionally, for any second sample image and any category, the sixth image features corresponding to that category satisfy the following relationship:

[0153]

[0154] Where, x t Let x be the second sample image of the target domain, and c be used to represent the second sample image x. t The soft-labeled image includes any one of at least one category. Used to represent the sixth image feature corresponding to category c Used to represent the sub-labeled image corresponding to category c. Used to represent sub-labeled images The number of pixels belonging to category c; f is used to represent pixel-wise multiplication. E (x t ) is used to represent the second sample image x t The fifth image feature, f E (·) is used to represent the feature extraction sub-model in the image annotation model.

[0155] Step 6038 is the same as step 6032 above, and will not be repeated here.

[0156] 604. The computer device clusters multiple first sample images based on the acquired multiple first image features to obtain at least one first cluster center.

[0157] In the embodiments of the present application, the first sample images of the source domain can include different scenes. For example, the first sample images of the source domain are all street view images, and part of the street view images are urban street view images, and the other part of the street view images are suburban street view images. The first image features of the plurality of first sample images of the source domain are clustered to obtain at least one cluster, that is, each cluster has a center, that is, at least one clustering center is obtained, and each first clustering center represents a scene of the source domain. For any first sample image, the scene contained in the first sample image is similar to the scene represented by the corresponding first clustering center. For example, one first clustering center is used to indicate suburban street view, and the first sample images belonging to the cluster corresponding to the first clustering center are all suburban street view images; another first clustering center is used to indicate urban street view, and the first sample images belonging to the cluster corresponding to the first clustering center are all urban street view images. Each first clustering center is equivalent to an anchor point of the source domain, and the plurality of first sample images of the source domain are distributed on the corresponding anchor points.

[0158] In the embodiments of the present application, the process of clustering the first sample images can use various clustering algorithms, such as k-means algorithm, hierarchical clustering algorithm or DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm.

[0159] In a possible implementation manner, the plurality of first image features are used for clustering to obtain at least one first clustering center of the plurality of first sample images, that is, the step 604 includes: selecting k first image features from the plurality of first image features as initial first clustering centers, determining the distance between each first sample image feature and each initial first clustering center, assigning each first sample image feature to the nearest initial first clustering center to obtain a plurality of clusters, determining the average value of the first sample image features in each cluster as an updated first clustering center, and then updating the first clustering center again based on the k updated first clustering centers in the above manner, and stopping updating the first clustering center in response to the iteration number reaching a threshold value or the k updated first clustering centers converging, to obtain the final k first clustering centers.

[0160] wherein k is any positive integer. The k first clustering centers are updated in an iterative manner, so that the obtained first clustering centers can represent the corresponding clusters, that is, the distance between each first clustering center and the first sample images in the corresponding cluster is the smallest, thereby ensuring the accuracy of the first clustering center.

[0161] 605、For any second sample image, the computer device determines distances between a second image feature of the second sample image and each first cluster center, determines a similarity between the second sample image and the first sample images based on a minimum distance in the determined distances, and the minimum distance corresponding to the second sample image is in a negative correlation with the similarity corresponding to the second sample image.

[0162] The distance between the second image feature and the first cluster center is a Euclidean distance, a Mahalanobis distance, or the like. In the embodiments of the present application, for any second sample image of the target domain, the distance between the second sample image and each first cluster center can reflect the similarity between the second sample image and the first sample images belonging to each first cluster center, and then the minimum distance can be determined from the determined distances, that is, the second sample image is most similar to the first sample images belonging to the first cluster center corresponding to the minimum distance, and then the similarity between the second sample image and the first sample images of the source domain is determined based on the minimum distance. Moreover, the similarity corresponding to any second sample image is in a negative correlation with the minimum distance corresponding to the second sample image, that is, the greater the minimum distance corresponding to the second sample image, the smaller the similarity corresponding to the second sample image, the smaller the minimum distance corresponding to the second sample image, and the greater the similarity corresponding to the second sample image.

[0163] The distance between the second sample image and each first cluster center can reflect the similarity between the second sample image and the first sample images belonging to each first cluster center, and then the similarity is determined based on the minimum distance corresponding to each second sample image, that is, the maximum similarity between the second sample image and the first sample images of the source domain is determined, thereby ensuring the accuracy of the determined similarity.

[0164] In a possible implementation manner, the minimum distance corresponding to any second sample image satisfies the following relationship:

[0165]

[0166] wherein, D(x t ) represents the minimum distance corresponding to the second sample image x t , F t (x t ) represents the second image feature of the second sample image x t , and ||·|| represents a norm.

[0167] ​606、The computer device selects a target sample image from the plurality of second sample images, acquires a second annotation image of the target sample image, and the target sample image is a second sample image having the smallest similarity between the target sample image and the sample image of the source domain in the plurality of second sample images.

[0168] The second annotation image is used to indicate the category to which each pixel point in the target sample image belongs, and the second annotation image is obtained by manual annotation.

[0169] Since the images of the target domain have unique information of the target domain, by selecting the target sample image having the smallest similarity with the first sample image of the source domain from the plurality of second sample images as much as possible, it is ensured that the image annotation model trained based on the target sample image can learn the unique information of the target domain, so as to improve the applicability of the trained image annotation model in the target domain, thereby improving the model performance of the image annotation model in the target domain. Figure 7 As shown in FIG. 6, the feature extraction sub-model in the image annotation model is called to acquire the first image feature of the first sample image and the second image feature of the second sample image, respectively, and the first sample image is clustered based on the first image feature to obtain the first clustering center, and the target sample image having the smallest similarity with the sample image of the source domain is selected from the plurality of second sample images based on the second image feature of the second sample image and the first clustering center.

[0170] In a possible implementation, when the target sample image is selected, a target number of target sample images are selected. The target number is an arbitrary value, for example, the target number is 10, or the target number is 5% of the total number of second sample images, etc. In the plurality of second sample images, the similarity corresponding to the target number of target sample images is smaller than the similarity corresponding to the remaining unselected second sample images.

[0171] In the embodiment of the present application, the first clustering center of the source domain is equivalent to the anchor point of the source domain, and the process of selecting the target sample image through the plurality of first clustering centers of the source domain is equivalent to the process of selecting the target sample image based on the plurality of anchor points of the source domain.

[0172] After the target sample image of the target domain is acquired, the image annotation model is iteratively trained multiple times, and the following steps 607-610 are only described by taking one iteration as an example.

[0173] 607. During the training of the image annotation model, the computer equipment calls the image annotation model to annotate the first sample image and the target sample image respectively, so as to obtain the first predicted annotation image of the first sample image and the second predicted annotation image of the target sample image.

[0174] In this embodiment, the image annotation model is the image annotation model trained according to step 302 above. The image annotation model is invoked to annotate the first sample image to obtain the first predicted annotation image of the first sample image. The image annotation model is then invoked to annotate the target sample image to obtain the second predicted annotation image of the target sample image.

[0175] In one possible implementation, the image annotation model includes a feature extraction sub-model and a feature transformation sub-model. The feature extraction sub-model and the feature transformation sub-model are used to annotate the image. That is, step 607 includes: calling the feature extraction sub-model to extract features from the first sample image and the target sample image respectively, to obtain the image features of the first sample image and the image features of the target sample image; and calling the feature transformation sub-model to transform the image features of the first sample image and the image features of the target sample image respectively, to obtain the first predicted annotation image of the first sample image and the second predicted annotation image of the target sample image.

[0176] 608. The computer device obtains at least one second cluster center corresponding to multiple second sample images in this iteration. The at least one second cluster center is obtained by clustering based on the second image features of the second sample images.

[0177] In this embodiment, during the iterative training of the image annotation model, the image annotation model is updated in each iteration. Therefore, the image annotation model may be different in different iterations. Since the second cluster center is obtained by clustering the second image features of the second sample image in the current iteration, and the second image features are obtained based on the image annotation model in the current iteration, the second image features and the second cluster center may change in each iteration. Therefore, each iteration needs to determine the second image features of the second sample image in the current iteration and at least one second cluster center in the current iteration.

[0178] In one possible implementation, when training the image annotation model through multiple iterations, the method for obtaining the second cluster center differs between the first iteration and other iterations. That is, the process of obtaining at least one second cluster center includes the following two methods:

[0179] The first mode: if the current iteration is the first iteration of the training process, the second image features of each second sample image in the current iteration are obtained; based on the obtained plurality of second image features, the plurality of second sample images are clustered to obtain at least one second cluster center.

[0180] This step is the same as step 604 described above, and will not be repeated here.

[0181] The second mode: if the current iteration is not the first iteration of the training process, each second image feature is assigned to the nearest second cluster center based on the distance between each second image feature and each second cluster center in the last iteration; each second cluster center is updated based on the corresponding second image feature of each second cluster center; the updated second cluster center is determined as the second cluster center corresponding to the current iteration.

[0182] According to the second image features of each second sample image in the last iteration and the at least one second cluster center, the second cluster center is updated, and the updated second cluster center is used as the second cluster center of the current iteration. This process does not need to re-cluster all second image features, reducing the amount of calculation, saving resources, improving the efficiency of obtaining the second cluster center, and ensuring the accuracy of the second cluster center.

[0183] In a possible implementation mode, the process of updating the second cluster center can satisfy the following relationship:

[0184]

[0185] Wherein, v is used to represent the vth second cluster center in the at least one second cluster center, v is a positive integer greater than 0; is used to represent the updated vth second cluster center, and a is used to represent an adjustment coefficient, a is a constant greater than 0 and less than 1, is used to represent the vth second cluster center in the last iteration, and F t (x t ) is used to represent the second image feature of the second sample image x t assigned to the vth second cluster center.

[0186] 609, the computer device obtains the distance between the second image feature of each second sample image in the current iteration and the at least one second cluster center.

[0187] In the embodiment of the present application, when training the image labeling model, the second image features of each second sample image need to be determined again in each iteration, and the determination manner is the same as that in step 603. When obtaining the second image features of each second sample image, for a target sample image in the plurality of second sample images, the second labeling image of the target sample image is used to obtain the second image features of the target sample image. Then, based on each second feature map and each second clustering center in the current iteration, the distance between each second image feature and at least one second clustering center is determined, that is, for any second image feature, the distance between the second image feature and each second clustering center is determined. The manner of determining the distance between the second image feature and the second clustering center is the same as that in step 605, and will not be described here.

[0188] 610, the computer device trains the image labeling model in the current iteration based on the difference between the first labeling image and the first predicted labeling image, the difference between the second labeling image and the second predicted labeling image, and the distance corresponding to each second sample image.

[0189] The training of the image labeling model in the current iteration based on the difference between the first labeling image and the first predicted labeling image, the difference between the second labeling image and the second predicted labeling image, and the distance corresponding to each second sample image is equivalent to adjusting the parameters of the image labeling model based on the difference between the first labeling image and the first predicted labeling image, the difference between the second labeling image and the second predicted labeling image, and the distance corresponding to each second sample image.

[0190] In the embodiment of the present application, the difference between the first labeling image and the first predicted labeling image, the difference between the second labeling image and the second predicted labeling image, and the distance corresponding to each second sample image, that is, the distance between each second sample image and each second clustering center, take into account the relationship between the second sample image and each second clustering center, so that the trained image labeling model can learn the image distribution of the target domain, thereby improving the accuracy of the image labeling model and the applicability of the image labeling model in the target domain.

[0191] In the embodiment of the present application, in the process of iteratively training the image labeling model according to steps 607-610, the second clustering center of the target domain is used in each iteration. The second clustering center is equivalent to an anchor point of the target domain, and the plurality of second sample images of the target domain are distributed on the corresponding anchor points. Therefore, the process of iteratively training the image labeling model through the plurality of second clustering centers of the target domain is equivalent to iteratively training the image labeling model based on the plurality of anchor points of the target domain, that is, training the image labeling model in a multi-anchor point based domain adaptation semantic segmentation method.

[0192] In a possible implementation, the image labeling model is trained based on the determined loss value, that is, the step 610 includes: determining a third loss value based on the difference between the first labeled image and the first predicted labeled image and the difference between the second labeled image and the second predicted labeled image, determining a fourth loss value based on the distance corresponding to each second sample image, and training the image labeling model of this iteration based on the sum of the third loss value and the fourth loss value.

[0193] The fourth loss value is obtained based on a distance loss function, the second clustering center is equivalent to an anchor point of the target domain, and the distance loss function is equivalent to a soft alignment loss function based on multiple anchor points.

[0194] The loss value is determined based on the difference between the first labeled image and the first predicted labeled image, the difference between the second labeled image and the second predicted labeled image, and the distance corresponding to each second sample image, so that the image labeling model can be trained according to the loss value, thereby ensuring the accuracy of training the image labeling model.

[0195] Optionally, the third loss value satisfies the following relationship:

[0196]

[0197] wherein L CE (x s , y s ) is used to represent the loss value, which is used to represent the difference between the first labeled image and the first predicted labeled image, x s is used to represent the first sample image, y s is used to represent the first labeled image, is used to represent the loss value, which is used to represent the difference between the second labeled image and the second predicted labeled image, is used to represent the target sample image, is used to represent the second labeled image of the target sample image.

[0198] Optionally, the fourth loss value satisfies the following relationship:

[0199]

[0200] wherein, is used to represent the fourth loss value, V is used to represent the number of second clustering centers, F t (x t ) is used to represent the second image feature of the second sample image x t in this iteration, is used to represent the υth second clustering center in this iteration, and ||·|| is used to represent a norm.

[0201] In a possible implementation, the image labeling model is trained by using the first sample image of the source domain, the target sample image of the target domain, and a third sample image of the target domain other than the target sample image, that is, the step 610 includes the following steps 6101-6103:

[0202] 6101, obtain a soft labeling image of the third sample image, the third sample image being a second sample image other than the target sample image in the plurality of second sample images, and the soft labeling image being obtained by labeling the third sample image by the image labeling model before training the image labeling model.

[0203] Any soft labeling image of a second sample image is used to indicate the category to which each pixel point in the second sample image belongs. In the embodiment of the present application, the plurality of second sample images of the target domain are divided into the target sample image and the third sample image, the target sample image has the second labeling image, and the third sample image does not correspond to a labeling image. Therefore, the third sample image is labeled by calling the image labeling model to obtain a labeling image of the third sample image, and the labeling image of the third sample image is determined as the soft labeling image of the third sample image.

[0204] In a possible implementation, before the step 607, the method further includes: calling the image labeling model to label the first sample image and the target sample image respectively to obtain a first predicted labeling image of the first sample image and a second predicted labeling image of the target sample image, and training the image labeling model based on the difference between the first labeling image and the first predicted labeling image and the difference between the second labeling image and the second predicted labeling image.

[0205] Since the obtained first sample image and the target sample image both correspond to a labeling image, the image labeling model is first trained by using the obtained first sample image, the first labeling image, the target sample image, and the second labeling image to improve the model performance of the image labeling model in the target domain, that is, to improve the applicability of the image labeling model in the target domain.

[0206] In the embodiment of the present application, after obtaining the image labeling model applicable to the source domain, the image labeling model is first trained according to the step 602, then steps 603-606 are executed, and then the image labeling model is trained based on the obtained first sample image, the first labeling image, the target sample image, and the second labeling image, and then the trained image labeling model is iteratively trained according to steps 607-610. The image labeling model is trained in this way to gradually improve the model performance of the image labeling model in the target domain, so as to improve the applicability of the image labeling model in the target domain.

[0207] In this embodiment of the application, the image annotation model used to obtain the soft-annotated image is an image annotation model that is first trained according to step 602 above, and then trained again based on the obtained first sample image, first annotated image, target sample image, and second annotated image. For example... Figure 8 As shown, the image annotation model is first trained according to step 602 above. Then, the image annotation model is trained based on the first sample image, the first annotation image, the target sample image, and the second annotation image. The trained image annotation model is used to annotate the third sample image to obtain the annotation image of the third sample image. The obtained annotation image of the third sample image is determined as the soft annotation image of the third sample image. Based on the feature extraction sub-model in the image annotation model, the image features of the target sample image and the image features of the third sample image are obtained, that is, the second image features of multiple second sample images are obtained. Clustering is performed based on the second image features of the second sample images to obtain at least one second cluster center.

[0208] 6102. Call the image annotation model to annotate the third sample image and obtain the third predicted annotation image of the third sample image.

[0209] 6103. Based on the differences between the first labeled image and the first predicted labeled image, the differences between the second labeled image and the second predicted labeled image, the distance corresponding to each second sample image, and the differences between the soft labeled image and the third predicted labeled image, the image labeling model for this iteration is trained.

[0210] The trained image annotation model is used to annotate images in the target domain.

[0211] Training soft-labeled images based on the distance to each second sample image results in image features output by the image annotation model that are close to the second cluster center, which leads to the loss of the unique information of the second sample images in the target domain. However, training the image annotation model by utilizing the difference between the soft-labeled image and the third predicted labeled image sharpens the image features output by the image annotation model, which can highlight the unique information of the sample images in the target domain, thereby improving the applicability of the image annotation model in the target domain and thus improving the model performance in the target domain.

[0212] In a possible implementation, the image labeling model is trained by using the determined loss value, that is, 6103 includes: determining a third loss value based on the difference between the first labeled image and the first predicted labeled image, the difference between the second labeled image and the second predicted labeled image, determining a fourth loss value based on the distance corresponding to each second sample image, determining a fifth loss value based on the difference between the soft labeled image and the third predicted labeled image, and training the image labeling model of this iteration based on the sum of the third loss value, the fourth loss value, and the fifth loss value.

[0213] The loss value is determined based on the difference between the first labeled image and the first predicted labeled image, the difference between the second labeled image and the second predicted labeled image, the distance corresponding to each second sample image, and the difference between the soft labeled image and the third predicted labeled image, so that the image labeling model can be trained according to the loss value, thereby ensuring the accuracy of training the image labeling model.

[0214] Optionally, the fifth loss value is obtained by using a pseudo-labeling loss function based on the difference between the soft labeled image and the third predicted labeled image.

[0215] Optionally, the fifth loss value satisfies the following relationship:

[0216]

[0217] wherein, L pseudo is used to represent the fifth loss value, L CE (·) is used to represent a loss function, is used to represent a target domain third sample image, is used to represent a third sample image of the soft labeled image.

[0218] Optionally, the sum of the third loss value, the fourth loss value, and the fifth loss value satisfies the following relationship:

[0219]

[0220] wherein, L semin is used to represent the sum of the third loss value, the fourth loss value, and the fifth loss value, L seg is used to represent the third loss value, is used to represent the fourth loss value, L pseudo is used to represent the fifth loss value.

[0221] As Figure 9As shown, the image labeling model comprises a feature extraction sub-model and a feature conversion sub-model, and the image labeling model is trained based on the first sample image, the first labeled image, the target sample image, the second labeled image, the soft labeled image of the third sample image, and the distance corresponding to each second sample.

[0222] In addition, the soft label image can comprise the category to which each pixel point in the corresponding third sample image belongs and the corresponding probability. When the image labeling model is trained based on the difference between the soft label image and the third predicted labeled image, only the pixel points with a probability greater than a threshold value are selected from the soft label image, and the image labeling model is trained based on the difference between the categories to which the pixel points with a probability greater than the threshold value belong in the soft label image and the third predicted labeled image. The threshold value is an arbitrary numerical value.

[0223] In addition, in the process of training the image labeling model, the labeled images obtained by labeling the same sample image based on the image labeling model in different iterations can have differences. When the image labeling model is trained based on the difference between the soft label image and the third predicted labeled image, the uncertain region of the third predicted labeled image is determined, and the difference between the region outside the uncertain region and the corresponding region in the soft label image is used to train the image labeling model. The uncertain region is used to represent the difference region between the labeled images obtained by labeling the same sample image multiple times by the image labeling model.

[0224] It should be noted that the embodiments of the present application are based on the differences between the first labeled image and the first predicted labeled image, the differences between the second labeled image and the second predicted labeled image, and the distance corresponding to each second sample image to train the image labeling model. In another embodiment, steps 608-610 do not need to be performed, and other ways can be adopted to train the image labeling model based on the differences between the first labeled image and the first predicted labeled image and the differences between the second labeled image and the second predicted labeled image.

[0225] In a possible implementation manner, in the process of iteratively training the image labeling model according to steps 607-610, if the number of iterations is greater than a threshold number of times, the training of the image labeling model is stopped, or if the sum of the third loss value, the fourth loss value, and the fifth loss value is less than a loss threshold value, the training of the image labeling model is stopped.

[0226] In the embodiment of the present application, the process of training the image labeling model can be implemented based on PyTorch (an open source code). According to a plurality of loss functions, the loss value of the image labeling model is determined, so as to train the image labeling model based on the determined loss value. And in the process of iteratively training the image labeling model, the parameters in the image labeling model are updated based on SGD (Stochastic Gradient Descent), and the initial learning rate of SGD is set to 2.5x10 -4 and the learning rate is gradually reduced using the Poly (decay) strategy with an energy of 0.9.

[0227] 611, the computer device calls the trained image labeling model to label the target image of the target domain, and obtains a labeled image of the target image.

[0228] This step is the same as step 607 described above, and will not be repeated here.

[0229] In the embodiment of the present application, after training the image labeling model, the trained image labeling model can be deployed in the blockchain, and any computer device in the blockchain can call the image labeling model to label the target image of the target domain.

[0230] The method provided in the embodiment of the present application, because there are differences between the images of the source domain and the target domain, the images of the target domain have unique information of the target domain, therefore, the sample image most dissimilar to the sample image of the source domain is selected from the target domain, and the selected sample image and the sample image and labeled image of the source domain are used to train the image labeling model, so that the image labeling model can learn the unique information of the sample image of the target domain, thereby improving the applicability of the image labeling model in the target domain, improving the model performance of the image labeling model in the target domain, and improving the labeling accuracy of the image labeling model in the target domain.

[0231] And there is no need to label each second sample image of the target domain, which reduces the cost of obtaining labeled images, thereby improving the efficiency of training the image labeling model.

[0232] The image labeling model obtained based on the image labeling method provided in the embodiment of the present application, compared with the image labeling models in the related art, as shown in Figure 10 , the image labeling model provided in the present application and the image labeling model 1 and the image labeling model 2 provided in the related art are used to label different original images, and the labeled images obtained are as shown in Figure 10 , Figure 10The last column of images is a manually annotated image, which is equivalent to a real annotated image. By comparing the annotated images corresponding to multiple image annotation models, it can be seen that the image annotation model provided in the application performs better on the target domain, and the annotated image obtained by the image annotation model provided in the application is closer to the real annotated image, that is, the accuracy of the annotated image obtained based on the image annotation model provided in the application is higher.

[0233] Take two semantic segmentation scenarios as examples, such as a game virtual scene-real scene and a synthesized virtual scene-real scene. The images in the virtual scene are taken as the source domain images, and the images in the real scene are taken as the target domain images. The image annotation model provided in the application and the image annotation model 1, the image annotation model 2, the image annotation model 3 and the image annotation model 4 provided by the related technology are used to annotate the images in different semantic segmentation scenarios. Table 2 is an example of a game virtual scene-real scene. As can be seen from Table 2, the annotation accuracy of the image annotation model provided in the application is higher. Table 3 is an example of a synthesized virtual scene-real scene. As can be seen from Table 3, the annotation accuracy of the image annotation model provided in the application is higher.

[0234] Table 2

[0235]

[0236] Table 3

[0237]

[0238]

[0239] Figure 11 is a structural schematic diagram of an image annotation device provided by an embodiment of the application, as shown in Figure 11 The device comprises:

[0240] The acquisition module 1101 is configured to acquire a first sample image of a source domain, a first annotated image of the first sample image, and a plurality of second sample images of a target domain.

[0241] The acquisition module 1101 is further configured to select a target sample image from the plurality of second sample images, and acquire a second annotated image of the target sample image. The target sample image is a second sample image in the plurality of second sample images that has the smallest similarity with the sample image of the source domain.

[0242] The annotation module 1102 is configured to call an image annotation model, and annotate the first sample image and the target sample image respectively to obtain a first predicted annotated image of the first sample image and a second predicted annotated image of the target sample image.

[0243] The training module 1103 is configured to train the image labeling model based on the difference between the first labeled image and the first predicted labeled image, and the difference between the second labeled image and the second predicted labeled image.

[0244] In a possible implementation, as shown in Figure 12 The apparatus further includes:

[0245] The acquisition module 1101 is further configured to acquire first image features of a plurality of first sample images and second image features of a plurality of second sample images.

[0246] The clustering module 1104 is configured to cluster the plurality of first sample images based on the acquired plurality of first image features, to obtain at least one first clustering center.

[0247] The determination module 1105 is configured to, for any second sample image, determine distances between the second image feature of the second sample image and each first clustering center, and determine a similarity between the second sample image and a sample image of the source domain based on a minimum distance in the determined distances, the minimum distance corresponding to the second sample image being in a negative correlation relationship with the corresponding similarity.

[0248] In another possible implementation, as shown in Figure 12 The acquisition module 1101 includes:

[0249] The feature extraction unit 1111 is configured to invoke a feature extraction sub-model in the image labeling model, and perform feature extraction on each first sample image to obtain a third image feature of each first sample image.

[0250] The fusion unit 1112 is configured to fuse the third image feature corresponding to each first sample image and the corresponding first labeled image to obtain the first image feature of each first sample image.

[0251] In another possible implementation, the first labeled image includes a region corresponding to at least one category; the fusion unit 1112 is configured to, for any first sample image, extract at least one sub-labeled image corresponding to each category from the first labeled image of the first sample image, the sub-labeled image being used to indicate a pixel point belonging to the corresponding category; fuse each sub-labeled image corresponding to each category and the third image feature of the first sample image to obtain a fourth image feature corresponding to each category; and splice the fourth image features corresponding to the at least one category to obtain the first image feature of the first sample image.

[0252] In another possible implementation, the fusion unit 1112 is configured to, for the sub-labeled image corresponding to any category, multiply the pixel value of each pixel point in the sub-labeled image by the corresponding feature value to obtain a product corresponding to each pixel point, and the feature value corresponding to any pixel point is a feature value in the third image feature that is located at the same position as the pixel point; and determine a ratio between the product corresponding to each pixel point and the number of pixel points, and the ratio corresponding to each pixel point constitutes the fourth image feature corresponding to the category, and the number of pixel points is the number of pixel points in the sub-labeled image that belong to the corresponding category.

[0253] In another possible implementation, as shown in Figure 12 the acquisition module 1101 includes:

[0254] The feature extraction unit 1111 is configured to call a feature extraction sub-model in the image labeling model to perform feature extraction on each second sample image to obtain a fifth image feature of each second sample image.

[0255] The first acquisition unit 1113 is configured to acquire a soft-labeled image of each second sample image, where the soft-labeled image is obtained by labeling the second sample image by the image labeling model before training the image labeling model.

[0256] The fusion unit 1112 is configured to fuse the fifth image feature of each second sample image and the corresponding soft-labeled image to obtain a second image feature of each second sample image.

[0257] In another possible implementation, as shown in Figure 12 the training module 1103 includes:

[0258] The second acquisition unit 1131 is configured to acquire at least one second clustering center corresponding to a plurality of second sample images in the current iteration in the process of training the image labeling model, and the at least one second clustering center is obtained based on the second image features of the second sample images.

[0259] The second acquisition unit 1131 is further configured to acquire a distance between the second image feature of each second sample image in the current iteration and the at least one second clustering center.

[0260] The training unit 1132 is configured to train the image labeling model in the current iteration based on the difference between the first labeled image and the first predicted labeled image, the difference between the second labeled image and the second predicted labeled image, and the distance corresponding to each second sample image.

[0261] In a possible implementation, the second obtaining unit 1131 is configured to: if the current iteration is the first iteration of the training process, obtain a second image feature of each second sample image in the current iteration; and perform clustering on the plurality of second sample images based on the plurality of obtained second image features, to obtain at least one second clustering center.

[0262] In a possible implementation, the second obtaining unit 1131 is further configured to: if the current iteration is not the first iteration of the training process, assign each second image feature to a second clustering center that is closest to the second image feature based on a distance between each second image feature and each second clustering center in a previous iteration; update each second clustering center based on the second image feature corresponding to the second clustering center; and determine the updated second clustering center as a second clustering center corresponding to the current iteration.

[0263] In a possible implementation, the obtaining module 1101 is further configured to obtain a soft-labeled image of a third sample image, the third sample image being a second sample image in the plurality of second sample images other than the target sample image, and the soft-labeled image being obtained by labeling the third sample image by the image labeling model before the image labeling model is trained.

[0264] The training unit 1132 is configured to call the image labeling model to label the third sample image, to obtain a third predicted-labeled image of the third sample image; and train the image labeling model of the current iteration based on a difference between the first labeled image and the first predicted-labeled image, a difference between the second labeled image and the second predicted-labeled image, the distance corresponding to each second sample image, and a difference between the soft-labeled image and the third predicted-labeled image.

[0265] In a possible implementation, the training module 1103 is further configured to perform adversarial training on the image labeling model and the discriminator model based on the first sample image, the first labeled image, and the plurality of second sample images, the discriminator model being configured to determine whether a labeled image output by the image labeling model is a labeled image of the first sample image.

[0266] In a possible implementation, the training module 1103 is configured to call the image labeling model to label the first sample image and the plurality of second sample images respectively, to obtain a fourth predicted-labeled image of the first sample image and a fifth predicted-labeled image of each second sample image; call the discriminator model to determine the fourth predicted-labeled image and the fifth predicted-labeled image, to obtain a determination result; and train the image labeling model and the discriminator model based on a difference between the fourth predicted-labeled image and the first labeled image and the determination result.

[0267] In another possible implementation manner, the labeling module 1102 is further configured to call the trained image labeling model to label the target image of the target domain to obtain a labeled image of the target image.

[0268] It should be noted that the image labeling apparatus provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above described functions. In addition, the image labeling apparatus and the image labeling method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be described here.

[0269] The computer device provided in the embodiments of the present application includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the image labeling method of the above embodiments.

[0270] Optionally, the computer device is provided as a terminal. Figure 13 A structural block diagram of a terminal 1300 provided in an example embodiment of the present application is shown. The terminal 1300 can be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal 1300 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0271] The terminal 1300 includes a processor 1301 and a memory 1302.

[0272] The processor 1301 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1301 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1301 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1301 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 1301 can also include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0273] The memory 1302 can include one or more computer-readable storage media that can be non-transitory. The memory 1302 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one computer program for being executed by the processor 1301 to implement the image labeling method provided by the method embodiments in the present application.

[0274] In some embodiments, the terminal 1300 can also optionally include a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302, and the peripheral device interface 1303 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1303 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, a positioning assembly 1308, and a power supply 1309.

[0275] The peripheral interface 1303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302 and the peripheral interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302 and the peripheral interface 1303 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0276] The radio frequency circuit 1304 is configured to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts electric signals into electromagnetic signals for transmission, or converts electromagnetic signals received into electric signals. Optionally, the radio frequency circuit 1304 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0277] The display screen 1305 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 is further configured to capture touch signals on or above the surface of the display screen 1305. The touch signals can be input to the processor 1301 as control signals for processing. In this case, the display screen 1305 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1305 can be one, disposed on the front panel of the terminal 1300; in other embodiments, the display screen 1305 can be at least two, respectively disposed on different surfaces of the terminal 1300 or in a folding design; in other embodiments, the display screen 1305 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 1300. Even, the display screen 1305 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0278] The camera assembly 1306 is configured to capture images or videos. Optionally, the camera assembly 1306 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1306 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0279] The audio circuit 1307 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1301 for processing, or input to the radio frequency circuit 1304 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 1300. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can the electrical signal be converted into a sound wave audible to humans, but also can be converted into a sound wave inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1307 can also include a headphone jack.

[0280] The positioning component 1308 is used to position the current geographic position of the terminal 1300 to realize navigation or LBS (Location Based Service). The positioning component 1308 can be a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China or the Galileo system of Russia.

[0281] The power supply 1309 is used to supply power to each component in the terminal 1300. The power supply 1309 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0282] In some embodiments, the terminal 1300 further includes one or more sensors 1310. The one or more sensors 1310 include but are not limited to: an acceleration sensor 1311, a gyroscope sensor 1312, a pressure sensor 1313, a fingerprint sensor 1314, an optical sensor 1315 and a proximity sensor 1316.

[0283] The acceleration sensor 1311 can detect the acceleration size in three coordinate axes of the coordinate system established by the terminal 1300. For example, the acceleration sensor 1311 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1301 can control the display screen 1305 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used for game or user motion data collection.

[0284] The gyroscope sensor 1312 can detect the body direction and rotation angle of the terminal 1300, and can collect 3D motions of the user on the terminal 1300 in cooperation with the acceleration sensor 1311. The processor 1301 can implement the following functions according to the data collected by the gyroscope sensor 1312: motion sensing (e.g., changing a UI according to a tilt operation of the user), image stabilization during shooting, game control, and inertial navigation.

[0285] The pressure sensor 1313 can be disposed on the side frame of the terminal 1300 and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is disposed on the side frame of the terminal 1300, the user's grip signal on the terminal 1300 can be detected, and left-hand or right-hand recognition or a shortcut operation can be performed by the processor 1301 according to the grip signal collected by the pressure sensor 1313. When the pressure sensor 1313 is disposed on the lower layer of the display screen 1305, the processor 1301 can control an operable control on a UI according to a pressure operation of the user on the display screen 1305. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0286] The fingerprint sensor 1314 is used to collect the fingerprint of the user, and the processor 1301 can identify the identity of the user according to the fingerprint collected by the fingerprint sensor 1314, or the fingerprint sensor 1314 can identify the identity of the user according to the collected fingerprint. When the identity of the user is identified as a trusted identity, the processor 1301 authorizes the user to perform a related sensitive operation, which includes unlocking the screen, viewing encrypted information, downloading software, payment, and changing settings, etc. The fingerprint sensor 1314 can be disposed on the front, back, or side of the terminal 1300. When a physical button or a manufacturer's logo is disposed on the terminal 1300, the fingerprint sensor 1314 can be integrated with the physical button or the manufacturer's logo.

[0287] The optical sensor 1315 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the display screen 1305 according to the ambient light intensity collected by the optical sensor 1315. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1305 is increased, and when the ambient light intensity is low, the display brightness of the display screen 1305 is decreased. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1315.

[0288] The proximity sensor 1316, also referred to as a distance sensor, is arranged on the front panel of the terminal 1300. The proximity sensor 1316 is configured to acquire the distance between the user and the front face of the terminal 1300. In an embodiment, when the proximity sensor 1316 detects that the distance between the user and the front face of the terminal 1300 gradually decreases, the display screen 1305 is switched from the bright screen state to the dark screen state under the control of the processor 1301; when the proximity sensor 1316 detects that the distance between the user and the front face of the terminal 1300 gradually increases, the display screen 1305 is switched from the dark screen state to the bright screen state under the control of the processor 1301.

[0289] Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the terminal 1300, and the terminal 1300 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components. Figure 13

[0290] Optionally, the computer device is provided as a server. Figure 14 FIG. 14 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 1400 can have a large difference due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1401 and one or more memories 1402. The memory 1402 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1401 to implement the method provided by the foregoing method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input / output interface, and other components for implementing the functions of the device, and details are not described herein.

[0291] The embodiment of the present application further provides a computer readable storage medium, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to implement the operations performed in the image labeling method of the foregoing embodiment.

[0292] The embodiment of the present application further provides a computer program product or a computer program, which includes computer program code stored in a computer readable storage medium. The processor of the computer device reads the computer program code from the computer readable storage medium, and the processor executes the computer program code, so that the computer device implements the operations performed in the image labeling method of the foregoing embodiment.

[0293] ​Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0294] The above only describes optional embodiments of the present application, and is not intended to limit the embodiments of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the present application.

Claims

1. An image labeling method, characterized by, The method comprises: obtaining a first sample image of a source domain, a first labeled image of the first sample image, and a plurality of second sample images of a target domain; selecting a target sample image from the plurality of second sample images, obtaining a second labeled image of the target sample image, the target sample image being a second sample image in the plurality of second sample images having the smallest similarity with the first sample image; calling an image labeling model to label the first sample image and the target sample image respectively to obtain a first predicted labeled image of the first sample image and a second predicted labeled image of the target sample image; training the image labeling model based on the difference between the first labeled image and the first predicted labeled image, and the difference between the second labeled image and the second predicted labeled image, the image labeling model being used to label images of the target domain.

2. The method of claim 1, wherein, Before the selecting a target sample image from the plurality of second sample images, the method further comprises: obtaining first image features of a plurality of the first sample images and second image features of the plurality of second sample images; clustering the plurality of first sample images based on the obtained first image features to obtain at least one first clustering center; for any second sample image, determining the distance between the second image feature of the second sample image and each first clustering center, and determining the similarity between the second sample image and the first sample image based on the smallest distance in the determined distances, the smallest distance corresponding to the similarity being in a negative correlation relationship.

3. The method of claim 2, wherein, The obtaining first image features of a plurality of the first sample images comprises: calling a feature extraction sub-model in the image labeling model to extract features from each of the first sample images to obtain third image features of each of the first sample images; fusing the third image features corresponding to each of the first sample images with the corresponding first labeled images to obtain first image features of each of the first sample images.

4. The method of claim 3, wherein, The first labeled image comprises a region corresponding to at least one category; and the fusing the third image features corresponding to each of the first sample images with the corresponding first labeled images to obtain first image features of each of the first sample images comprises: for any first sample image, extracting a sub-labeled image corresponding to the at least one category from the first labeled image of the first sample image, the sub-labeled image being used to indicate pixel points belonging to the corresponding category; fusing the sub-labeled image corresponding to each category with the third image features of the first sample image to obtain fourth image features corresponding to each category; splicing the fourth image features corresponding to the at least one category to obtain the first image features of the first sample image.

5. The method of claim 4, wherein, The fusing the sub-labeled image corresponding to each category with the third image features of the first sample image to obtain fourth image features corresponding to each category comprises: For the sub-labeled image corresponding to any category, the pixel value of each pixel point in the sub-labeled image is multiplied by the corresponding feature value to obtain the product corresponding to each pixel point. The feature value corresponding to any pixel point is the feature value in the third image feature corresponding to the pixel point located at the same position; The ratio between the product corresponding to each pixel point and the number of pixel points is determined, and the ratio corresponding to each pixel point constitutes the fourth image feature corresponding to the category. The number of pixel points is the number of pixel points in the sub-labeled image belonging to the corresponding category.

6. The method of claim 2, wherein, Obtaining the second image features of the plurality of second sample images comprises: Calling the feature extraction sub-model in the image labeling model, performing feature extraction on each second sample image to obtain the fifth image feature of each second sample image; Obtaining the soft-labeled image of each second sample image, the soft-labeled image being obtained by labeling the second sample image by the image labeling model before training the image labeling model; Fusing the fifth image feature of each second sample image with the corresponding soft-labeled image to obtain the second image feature of each second sample image.

7. The method of claim 1, wherein, The training of the image labeling model based on the differences between the first labeled image and the first predicted labeled image, and the differences between the second labeled image and the second predicted labeled image comprises: During the training of the image labeling model, at least one second clustering center corresponding to the plurality of second sample images in the current iteration is obtained, and the at least one second clustering center is obtained based on the second image features of the second sample images. Obtaining the distance between the second image feature of each second sample image in the current iteration and the at least one second clustering center; Training the image labeling model of the current iteration based on the differences between the first labeled image and the first predicted labeled image, the differences between the second labeled image and the second predicted labeled image, and the distance corresponding to each second sample image.

8. The method of claim 7, wherein, The obtaining of the at least one second clustering center corresponding to the plurality of second sample images in the current iteration comprises: If the current iteration is the first iteration of the training process, obtaining the second image feature of each second sample image in the current iteration; Based on the obtained plurality of second image features, clustering the plurality of second sample images to obtain the at least one second clustering center.

9. The method of claim 8, wherein, The obtaining of the at least one second clustering center corresponding to the plurality of second sample images in the current iteration further comprises: If the current iteration is not the first iteration of the training process, assigning each second image feature to the nearest second clustering center based on the distance between each second image feature and each second clustering center in the last iteration; Updating each second clustering center based on the second image feature corresponding to each second clustering center, respectively; Determining the updated second clustering center as the second clustering center corresponding to the current iteration.

10. The method of claim 7, wherein, The method further comprises: obtain a soft-labeled image of a third sample image, the third sample image being a second sample image in the plurality of second sample images other than the target sample image, the soft-labeled image being obtained by labeling the third sample image by the image labeling model before training the image labeling model; the training of the image labeling model in the current iteration based on the differences between the first labeled image and the first predicted labeled image, the differences between the second labeled image and the second predicted labeled image, and the distance corresponding to each second sample image comprises: calling the image labeling model to label the third sample image to obtain a third predicted labeled image of the third sample image; the training of the image labeling model in the current iteration based on the differences between the first labeled image and the first predicted labeled image, the differences between the second labeled image and the second predicted labeled image, the distance corresponding to each second sample image, and the differences between the soft-labeled image and the third predicted labeled image.

11. The method of claim 1, wherein, Before the training of the image labeling model based on the differences between the first labeled image and the first predicted labeled image and the differences between the second labeled image and the second predicted labeled image, the method further comprises: performing adversarial training of the image labeling model and a discriminator model based on the first sample image, the first labeled image, and the plurality of second sample images, the discriminator model being configured to determine whether a labeled image output by the image labeling model is a labeled image of the first sample image.

12. The method of claim 11, wherein, The adversarial training of the image labeling model and the discriminator model based on the first sample image, the first labeled image, and the plurality of second sample images comprises: calling the image labeling model to label the first sample image and the plurality of second sample images respectively to obtain a fourth predicted labeled image of the first sample image and a fifth predicted labeled image of each second sample image; calling the discriminator model to determine the fourth predicted labeled image and the fifth predicted labeled image to obtain a determination result; training the image labeling model and the discriminator model based on the differences between the fourth predicted labeled image and the first labeled image and the determination result.

13. An image labeling apparatus characterized by comprising: The apparatus comprises: an obtaining module configured to obtain a first sample image of a source domain, a first labeled image of the first sample image, and a plurality of second sample images of a target domain; the obtaining module is further configured to select a target sample image from the plurality of second sample images, and obtain a second labeled image of the target sample image, the target sample image being a second sample image in the plurality of second sample images that has the smallest similarity with the first sample image; a labeling module configured to call an image labeling model to label the first sample image and the target sample image respectively to obtain a first predicted labeled image of the first sample image and a second predicted labeled image of the target sample image; The training module is configured to train the image annotation model based on differences between the first annotated image and the first predicted annotated image and differences between the second annotated image and the second predicted annotated image, the image annotation model being configured to annotate images of the target domain.

14. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement operations performed in the image annotation method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement operations performed in the image annotation method according to any one of claims 1 to 12.

Images