Image processing method and apparatus

Abstract

This application proposes an image processing method and apparatus, applicable to the field of image processing. The method includes: acquiring a set of sample images for model training; performing image transformation processing on the sample image set to obtain an auxiliary image set; selecting a training dataset for a target task from the sample image set and the auxiliary image set, and training a first model using the target task training dataset; selecting a training dataset for an auxiliary task of the target task from the sample image set and the auxiliary image set, and training a second model using the auxiliary task training dataset, wherein the second model shares the model structure and model parameters of the feature extractor with the first model; and determining the trained first model as the target model for performing the target task. This application can improve the accuracy of image recognition.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to the field of image processing technology, and in particular to image processing methods and apparatus. Background Technology

[0002] With the continuous development and evolution of deep learning, neural network models have been widely used in the field of image processing, such as neural network image recognition.

[0003] Currently, image recognition methods primarily rely on deep learning-based classification approaches. This involves training a deep learning-based classification model and directly determining the image type. In existing solutions, the sample images used for model training are typically collected manually and labeled manually. Because the number of labeled sample images is limited, only a small portion can be used for model training, resulting in inaccurate recognition performance from models trained on a small number of samples. Summary of the Invention

[0004] This application provides an image processing method, apparatus, computer device, and storage medium, which can train a first model and a second model based on a sample image set and an auxiliary image set, respectively, to improve the accuracy of image recognition.

[0005] On one hand, embodiments of this application provide an image processing method, including:

[0006] Obtain a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes first sample images and their category labels. The second set of sample images includes second sample images. Both the first and second sample images are images from the target domain corresponding to the target task.

[0007] The sample image set is subjected to image transformation processing to obtain an auxiliary image set; the auxiliary image set includes at least a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained by image enhancement processing of the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image;

[0008] The training dataset for the target task is selected from the sample image set and the auxiliary image set, and the first model is trained using the training dataset for the target task; the training dataset for the target task includes at least the first sample image set and the first augmented image set;

[0009] The training dataset for the auxiliary task of the target task is selected from the sample image set and the auxiliary image set, and the second model is trained using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model;

[0010] The trained first model is designated as the target model for performing the target task. The target model is used to identify images in the target domain.

[0011] On one hand, embodiments of this application provide an image processing apparatus, including:

[0012] The acquisition unit is used to acquire a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and a category label for the first sample image. The second set of sample images includes a second sample image. Both the first and second sample images are images in the target domain corresponding to the target task.

[0013] The processing unit is used to perform image transformation processing on the sample image set to obtain an auxiliary image set; the auxiliary image set includes at least a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained after image enhancement processing on the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image;

[0014] The training unit is used to select a training dataset for the target task from a set of sample images and a set of auxiliary images, and to train a first model using the training dataset for the target task; the training dataset for the target task includes at least a first set of sample images and a first set of augmented images.

[0015] The training unit is used to select the training dataset for the auxiliary task of the target task from the sample image set and the auxiliary image set, and to train the second model using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model;

[0016] The determination unit is used to identify the trained first model as the target model for performing the target task. The target model is used to recognize images in the target domain.

[0017] On one hand, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the image processing method described above.

[0018] On one hand, embodiments of this application provide a computer storage medium storing a computer program, which includes program instructions. When the program instructions are executed by a processor, they perform the image processing method described above.

[0019] One aspect of this application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer instructions are executed by the processor of a terminal device, the methods described in the above embodiments are performed.

[0020] The image processing method provided in this application embodiment can perform image transformation processing on the sample image set used for model training to obtain an auxiliary image set. Model training is then performed using both the sample image set and the auxiliary image set, thus solving the problem of training data collection, enriching the training data for model training, and facilitating the training of a more accurate model. Furthermore, during the training of the first model, a second model is selected to assist in training. This second model can share the model structure and model parameters of the feature extractor with the first model. Training data is selectively chosen from the sample image set and the auxiliary image set based on the first and second models. This training data can include labeled or unlabeled images. Through this model structure and self-supervised / semi-supervised training methods, the trained first model can achieve better performance. Therefore, when using the trained first model to perform target task-oriented image recognition in the target domain, superior image recognition results can be obtained, thereby improving the accuracy of image recognition. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0023] Figure 2a This is a scene illustration of a sample image set provided in an embodiment of this application;

[0024] Figure 2b This is a schematic diagram of a scenario for training a first model provided in an embodiment of this application;

[0025] Figure 2c This is a schematic diagram of a scenario for training a second model provided in an embodiment of this application;

[0026] Figure 2d This is a schematic diagram of another scenario for training a second model provided in an embodiment of this application;

[0027] Figure 2e This is a schematic diagram of an image processing scenario provided in an embodiment of this application;

[0028] Figure 2f This is a schematic diagram of another image processing scenario provided in an embodiment of this application;

[0029] Figure 3 This is a schematic flowchart of an image processing method provided in the application embodiment;

[0030] Figure 4a This is a schematic diagram of an image segmentation process provided in an embodiment of this application;

[0031] Figure 4b This is a schematic diagram of another image segmentation process provided in an embodiment of this application;

[0032] Figure 4c This is a schematic diagram of a process for training a first model provided in an embodiment of this application;

[0033] Figure 5 This is a flowchart illustrating another image processing method provided in the application embodiment;

[0034] Figure 6a This is a schematic diagram of a process for training a first auxiliary model provided in an embodiment of this application;

[0035] Figure 6b This is a schematic diagram of a process for training a second auxiliary model provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0040] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] The solutions provided in this application belong to the fields of computer vision and deep learning technologies within the field of artificial intelligence.

[0042] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0043] Deep Learning (DL) is an interdisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0044] This application primarily relates to image processing using computer vision techniques on a sample image set. Specifically, it involves image transformation processing of the sample image set to obtain an auxiliary image set; and it also relates to training a first model and a second model using deep learning techniques. Subsequently, when both the first and second models satisfy the model convergence condition, the trained first model is used as the target model corresponding to the target task, and the target model can be invoked to perform recognition processing on the image to be processed, thereby obtaining the image category of the image in the target domain.

[0045] This application can be applied to the following scenarios: A set of sample images for model training can be obtained; a first model and a second model can be trained using the sample image set and an auxiliary image set obtained after image transformation processing of the sample image set; and the trained first model can be used as the target model. The target model can be a model for recognizing human images, a model for recognizing medical images of cancer cells, or a model for recognizing animal categories, etc.

[0046] If the target model is designed for recognizing images in the portrait domain, and the image to be processed is also within the portrait domain, then when image recognition of the image to be processed is required, the target model trained in this application can be invoked to identify the image and determine its category within the portrait domain. For example, it can determine whether the image to be processed is a non-mainstream image. Subsequently, in video review systems and video copyright recognition systems, if the image to be processed is identified as a non-mainstream image, it can alert reviewers to conduct focused review or directly block the image.

[0047] If the target model is a medical image model for identifying cancer cells, and the image to be processed is an image within the medical field, then when image recognition of the image to be processed is required, the target model trained in this application can be called to identify the image to be processed and determine the image recognition of the image to be processed within the medical field, for example, to identify whether the image to be processed contains cancer cells.

[0048] If the target model is a model for identifying animal domain images and the image to be processed is an image within the animal domain, then when image recognition of the image to be processed is required, the target model trained in this application can be called to identify the image to be processed and determine the image recognition of the image to be processed within the animal domain, for example, to identify which animal category the image to be processed belongs to (cat, dog, rabbit, etc.).

[0049] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application. The image processing system includes a server 140 and a cluster of terminal devices, wherein the cluster of terminal devices may include terminal devices 110, 120, ..., 130, etc. The cluster of terminal devices and the server 140 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0050] Figure 1 The server 140 shown can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0051] Figure 1 The terminal devices 110, 120, and 130 shown can be mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), vehicles, roadside devices, aircraft, wearable devices such as smartwatches, smart bracelets, pedometers, and other smart devices with image processing capabilities.

[0052] Taking terminal device 110 as an example, terminal device 110 acquires a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and its category label. The second set of sample images includes a second sample image. Both the first and second sample images are images from the target domain corresponding to the target task. Terminal device 110 can send the set of sample images to server 140. Server 140 performs image transformation processing on the set of sample images to obtain an auxiliary image set. The auxiliary image set includes a first augmented image set, which includes a first augmented image and its category label. The first augmented image is obtained by enhancing the first sample image, and its category label is consistent with that of the first sample image. Server 140 selects a training dataset for the target task from a set of sample images and a set of auxiliary images, and trains a first model using the training dataset for the target task; the training dataset for the target task includes at least a first set of sample images and a first set of augmented images; and server 140 selects a training dataset for the auxiliary task of the target task from the set of sample images and the set of auxiliary images, and trains a second model using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model; the trained first model is determined as the target model for performing the target task, wherein the target model can be used to recognize images in the target domain.

[0053] Server 140 sends the trained target model to terminal device 110. Subsequently, when terminal device 110 receives an image recognition request submitted by the user, the image recognition request carries the image to be processed in the target domain. Then, terminal device 110 can call the target model to perform image recognition on the image to be processed and determine the image category of the image to be processed.

[0054] In one possible implementation, when terminal device 110 receives an image recognition request submitted by a user, the request carries the image to be processed within the target domain. Terminal device 110 then sends the image to be processed to server 140. Server 140 invokes a target model to perform image recognition on the image to determine its image category. Finally, server 140 sends the image category of the image to be processed back to terminal device 110. Subsequently, terminal device 110 can display the image category of the image to be processed on the image recognition interface.

[0055] It should be noted that the sample image set undergoes image transformation processing to obtain an auxiliary image set; a training dataset for the target task is selected from the sample image set and the auxiliary image set, and the first model is trained using the target task training dataset; a training dataset for the auxiliary task of the target task is selected from the sample image set and the auxiliary image set, and the second model is trained using the auxiliary task training dataset; finally, all the operation steps of the trained target model are not necessarily executed by server 140. They can also be executed by terminal device 110 or any terminal device in the terminal device cluster.

[0056] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0057] In this embodiment, the first model refers to the model used to perform the target task (or main task), that is, the trained first model needs to be used to perform the target task; for example, if the target task is a face recognition task, then the trained first model is the model used to perform face recognition. The second model refers to an auxiliary model used to assist the training of the first model. The first model and the second model share the model structure and model parameters of the feature extractor, which allows the first model to perform the auxiliary task simultaneously; for example, if the target task is face recognition and the auxiliary task is to identify the position of the left eye in a face, then the second model can be an auxiliary model that can identify the position of objects in an image (e.g., a position-based classification model). The model structure and model parameters of the feature extractor are updated through the training of the second model and shared with the first model. This allows the feature extractor of the trained first model to have the ability to extract features for performing the target task as well as features for performing the auxiliary task, thereby enabling the trained first model to perform the auxiliary task while performing the target task, thus improving the capabilities of the first model.

[0058] The second model is used to assist the first model in training; therefore, the second model can also be called an auxiliary model. In this embodiment, both the first model and the second model can refer to models containing feature extractors, and the first model and the second model share the model structure and model parameters of the feature extractor. It should be specifically noted that sharing the model structure of the feature extractor between the first and second models means that the feature extractor in the first model and the feature extractor in the second model have the same and consistent model structure. When the model structure of either the feature extractor in the first model or the feature extractor in the second model changes, the model structure of the other changes accordingly. Sharing the model parameters of the feature extractor between the first and second models means that the feature extractor in the first model and the feature extractor in the second model maintain the same model parameters. When the model parameters of the feature extractor in the first model are updated after one or more training iterations, the model parameters of the feature extractor in the second model are updated accordingly; similarly, when the model parameters of the feature extractor in the second model are updated after one or more training iterations, the model parameters of the feature extractor in the first model are updated accordingly.

[0059] Understandably, the number and structure of the second auxiliary model are determined based on the actual auxiliary task. For example, in the above example, if the auxiliary task requires identifying the location of an object in an image, then the second auxiliary model can use a first auxiliary model, which could be a location-based classification model. Similarly, if the auxiliary task requires identifying the feature similarity between images, then the second auxiliary model can use a second auxiliary model, which could be a feature similarity-based discriminative model. Furthermore, if the auxiliary task simultaneously requires identifying the location of an object in an image and identifying the feature similarity between images, then the second auxiliary model can use both a first auxiliary model and a second auxiliary model.

[0060] The image processing flow of the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0061] Please see Figure 2a , Figure 2aThis is a schematic diagram of a sample image set provided in an embodiment of this application. Users can obtain sample data (sample image set) for model training online through data crawling. The sample image set may include a first sample image set and a second sample image set. The first sample image set includes first sample images and their category labels; the second sample image set includes second sample images. Both the first and second sample images are images from the target domain corresponding to the target task. It should be noted that the first sample image refers to an image with a category label, and the number of first sample images in the first sample image set can be one or more; this application does not limit the number. The second sample image refers to an image without a category label; similarly, the number of second sample images in the second sample image set can be one or more; this application also does not limit the number. For example, such as... Figure 2a As shown, the target domain is the portrait domain, and the target task is to identify the image category corresponding to images belonging to the portrait domain. Specifically, it is to identify whether an image is a mainstream image or a non-mainstream image. Non-mainstream images refer to portraits with features such as exaggerated hairstyles, bright hair colors, and a disheveled appearance. Therefore, the first sample image set can be a set of images containing both mainstream and non-mainstream images (both mainstream and non-mainstream images carry category labels), and the second sample image set can be a set of unlabeled portraits.

[0062] In addition to images within the target domain, the sample image set may also include images outside the target domain, referred to as third sample images. For example, in the above example, if the target domain is the human portrait domain, then the third sample images could be non-human portrait images, such as images of objects, anime, or landscapes. That is, the sample image set may also contain a third sample image set, which includes the third sample images and their category labels (as in the example above, the category label for non-human portrait images is "non-human portrait"). Similarly, the number of third sample images in the third sample image set can be one or more, and this application does not limit the number.

[0063] In one possible implementation, a computer device can perform image transformation processing on a set of sample images to obtain an auxiliary image set. The auxiliary image set includes at least a first augmented image set, which comprises a first augmented image and its category label. The first augmented image is obtained by enhancing the first sample image, and it also includes a category label, which is consistent with the category label of the first sample image. Specifically, the image transformation processing includes, but is not limited to, image enhancement processing and image segmentation processing. Image enhancement processing can include, for example, random cropping; horizontal image flipping; image rotation; adding noise; modifying image brightness; modifying image saturation; covering a small portion of image content; and Gaussian blurring. Furthermore, the above methods can be combined, for example, rotating the image first and then applying Gaussian blur, etc. Image segmentation processing can include, for example, dividing the image into thirds, fourths, ninths, etc.

[0064] Please see Figure 2b , Figure 2b This is a schematic diagram illustrating a scenario for training a first model according to an embodiment of this application. In one possible implementation, the computer device selects a training dataset for the target task from a sample image set and an auxiliary image set, and trains the first model using the training dataset for the target task. The model structure of the first model can be found in [reference needed]. Figure 2b As shown, the first model includes a first feature extractor and a first classifier connected to the first feature extractor. Furthermore, the training dataset for the target task includes at least a first set of sample images and a first set of augmented images. For example, as... Figure 2bAs shown, the first sample image from the first sample image set and the first augmented image from the first augmented image set can be input into the first model as the training dataset for the first model. If the first sample image is an image in the portrait domain, and its category label is mainstream image or non-mainstream image, then after image augmentation processing on the mainstream and non-mainstream images, the first augmented image set is obtained. The first augmented images in the first augmented image set will also carry category labels, and the category labels of the first augmented images are consistent with the category labels of the first sample images. For example, if the category label of a certain first sample image is mainstream image, then the category label of the corresponding first augmented image is also mainstream image; if the category label of a certain first sample image is non-mainstream image, then the category label of the corresponding first augmented image is also non-mainstream image. Since each image in the first sample image set and the first augmented image set carries a label (mainstream or non-mainstream), the first model (including the first feature extractor and the fully connected layer FC1) is used, where the fully connected layer FC1 is the first classifier. The first model learns the category labels of images; therefore, the corresponding classification result in the first model is either non-mainstream or mainstream. As described above, the first model is trained using a first set of sample images and a first set of augmented images until it reaches a convergence condition, at which point training stops. Here, the convergence condition can be any of the following: the loss function of the first model is less than a set threshold; the loss function of the first model has stabilized and no longer changes as training continues; all training data used to train the first model (e.g., each image in the first set of sample images and the first set of augmented images) has participated in the training; the number of training iterations of the first model reaches a reference training quantity threshold (wherein this reference training quantity threshold is much smaller than the training threshold that is usually required for model training), etc.

[0065] In one possible implementation, the computer device selects a training dataset for an auxiliary task from a set of sample images and a set of auxiliary images, and trains a second model using the training dataset for the auxiliary task. See also... Figure 2c , Figure 2c This is a schematic diagram of a scenario for training a second model according to an embodiment of this application. In one embodiment, the auxiliary task may be to identify the location of an object in an image. In this embodiment, the model structure of the second model can be found in [reference needed]. Figure 2c ,like Figure 2cThe second model shown may include a second feature extractor and a second classifier connected to the second feature extractor. Furthermore, the second model shares the model structure and model parameters of the feature extractor with the first model; specifically, the first feature extractor and the second feature extractor share the model structure and model parameters. In this embodiment, the auxiliary image set may also include a first segmented image set and a second segmented image set. The first segmented image set contains a first segmented image and its location labels; the first segmented image is obtained by segmenting the first sample image. The second segmented image set contains a second segmented image and its location labels; the second segmented image is obtained by segmenting the second sample image. Therefore, the training dataset for the auxiliary task here includes the first segmented image set and the second segmented image set. The second model (including the second feature extractor and a fully connected layer FC2) is used, where the fully connected layer FC2 is the second classifier. The classification result corresponding to the second model can be the location labels in the image. Similarly, when the second model reaches the convergence condition, training of the second model stops. Here, the convergence condition of the second model can be any of the following: the loss function of the second model is less than a set threshold; the loss function of the second model has stabilized and no longer changes as the training process continues; all training data used to train the second model has been used in the training; the number of training iterations of the second model reaches a reference training quantity threshold (wherein, the reference training quantity threshold is much smaller than the training threshold that usually needs to be met when training a model), etc.

[0066] In another implementation, the auxiliary task could be to identify feature similarities between images. See also Figure 2d , Figure 2d This is a schematic diagram illustrating another scenario for training the second model provided in this application embodiment. The model structure of the second model in this embodiment can be found in [reference needed]. Figure 2d ,like Figure 2dThe second model shown may include two third feature extractors and similarity discriminators connected to the two third feature extractors respectively. The Siamese network model used here has identical model structures and parameters between the two third feature extractors. Furthermore, the second model shares the model structure and parameters of the feature extractors with the first model; specifically, the first feature extractor and the two third feature extractors share the same model structure and parameters. In this embodiment, the auxiliary image set may include: a first segmented image set, a second segmented image set, a first augmented image set, and a second augmented image set. The first segmented image set contains the first segmented image and its location label; the first segmented image is obtained by segmenting the first sample image. The second segmented image set contains the second segmented image and its location label; the second segmented image is obtained by segmenting the second sample image. The second augmented image set contains the second augmented image, which is obtained by enhancing the second sample image. Therefore, the training dataset for the auxiliary task here includes a first set of segmented images, a second set of segmented images, a first set of augmented images, a second set of augmented images, and a set of sample images. The classification result corresponding to the second model can be the feature similarity between images. Similarly, training of the second model stops when it reaches the convergence condition. Here, the convergence condition of the second model can be any of the following: the loss function of the second model is less than a set threshold; the loss function of the second model has stabilized and no longer changes as the training process continues; all training data used to train the second model has participated in the training; the number of training iterations of the second model reaches a reference training quantity threshold (wherein, the reference training quantity threshold is much smaller than the training threshold that is usually required when training a model), etc.

[0067] In this embodiment, a joint training process of the first model and the second model includes: ① Training the first model first and updating the model parameters of the first feature extractor; ② Simultaneously updating the model parameters of the second feature extractor based on the updated model parameters of the first feature extractor; ③ Training the second model and updating the model parameters of the second feature extractor; ④ Conversely, simultaneously updating the model parameters of the first feature extractor based on the updated model parameters of the second feature extractor. This completes one joint training session, which can be called a loop process. Repeating the above loop process allows for joint alternating training of the first and second models. Of course, the above joint training process can also involve training the second model first and then the first model; this application does not limit this training order. Regardless of which model is trained first, the feature extractor used during the training of the other model is the feature extractor of the previously trained model, so that the feature extractor can simultaneously learn the model performance required by both models. Once both the first model and the second model reach convergence, the training of the first model and the second model is complete.

[0068] Subsequently, the trained first model can be used as the target model for the target task, where the target model is used to recognize images in the target domain. For an example, please refer to... Figure 2e , Figure 2e This is a schematic diagram of an image processing scenario provided by an embodiment of this application. If the target domain is the human image domain, this application can be applied to video review systems, video copyright recognition systems, and as a preliminary filtering judgment for non-mainstream videos. Specifically, this application can identify whether a target image or video contains non-mainstream images or videos, automatically provide a discrimination label, and remind reviewers to focus on reviewing the video or directly block it. Figure 2e As shown in the middle left figure (10), if a user selects an image to be processed in their album and uploads it to a website or platform, then the website or platform can use the solution of this application to identify the image category of the image to be processed. Specifically, it can determine whether the image to be processed belongs to a non-mainstream image. Non-mainstream images specifically refer to images with exaggerated hair styles, bright hair colors, decadent styles, etc. Figure 2e As shown in the middle right figure (20), if the image to be processed is a non-mainstream image, the system can display a prompt pop-up window, which includes an "intercept" control and an "exit" control. The user (website operation and maintenance personnel) can click the "intercept" control, and the system can intercept and filter the image to be processed. This solution is conducive to maintaining a harmonious and healthy website or platform.

[0069] Of course, besides users manually selecting to block and filter non-mainstream images, the system can also automatically filter and process images after they are identified as non-mainstream. Please see [link / reference]. Figure 2f , Figure 2f This is a schematic diagram of another image processing scenario provided in an embodiment of this application. For example... Figure 2f As shown in the left-middle figure (30), the user uploads an image to be processed. Then, using the solution of this application, if the image to be processed is identified as a non-mainstream image, the system automatically filters the non-mainstream image and displays a prompt interface. The prompt interface may include a prompt message, which may be: "Non-mainstream image, automatically filtered." Through this solution, the system can automatically identify the image category of the image to be processed and automatically filter non-mainstream images. This can automatically filter the image to be processed, reduce the workload of website operation and maintenance personnel, and facilitate the creation of a good network atmosphere.

[0070] The image processing method provided in this solution, by using a second model that performs an auxiliary task to assist in the training of a first model that performs the target task, enables the finally trained target model (the trained first model) to obtain superior image recognition results when performing the target task to recognize images in the target domain. In various business scenarios involving image recognition, the target model provided in this application can better recognize images and improve the accuracy of image recognition.

[0071] Please see Figure 3 , Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of the application. The method is applied to computer devices, such as... Figure 3 As shown, the data processing method may include steps S310 to S350. Wherein:

[0072] Step S310: Obtain a set of sample images for model training, the set of sample images including a first set of sample images and a second set of sample images.

[0073] In specific implementation, the first sample image set includes first sample images and their category labels, and the second sample image set includes second sample images. Both the first and second sample images are images within the target domain corresponding to the target task. It should be noted that the first sample image refers to an image with a category label, and the number of first sample images in the first sample image set can be one or more; this application does not limit their number. The second sample image refers to an image without a category label; similarly, the number of second sample images in the second sample image set can be one or more, and this application also does not limit their number. The target domain is the portrait domain, and the target task is to identify the image category corresponding to images belonging to the portrait domain, specifically identifying whether an image is a mainstream image or a non-mainstream image. Non-mainstream images refer to portraits with features such as exaggerated hairstyles, bright hair colors, and a disheveled appearance. Therefore, the first sample image set can be a set of images containing both mainstream and non-mainstream images (both mainstream and non-mainstream images are labeled), and the second sample image set can be a set of unlabeled portraits.

[0074] In one possible implementation, the computer device can acquire the set of sample images for model training by online data crawling. Specifically, data crawling can include: (1) crawling non-mainstream image data through web crawlers, such as websites with publicly available images like Google, Baidu, and Yahoo, using various keywords related to faces, hair, lips, etc.; (2) using some large image classification datasets for subset parsing and crawling, such as Google's large open-source dataset OpenImage, to filter out data related to face images.

[0075] Furthermore, the sample image set can be crawled online, and the sample data can be cleaned. Since data cleaning is very labor-intensive, it is not necessary to process all collected training data; the uncleaned data is considered unlabeled. It should be noted that data cleaning is a process of re-examining and validating data, aiming to remove duplicate information, correct existing errors, and ensure data consistency.

[0076] The target domain can include human imagery, medical imagery, animal imagery, and natural imagery, among others. If the target domain is human imagery, the objective task can be to identify the image category corresponding to an image within that domain, specifically whether the image is a mainstream or non-mainstream image.

[0077] The first sample image can carry the category label as either "mainstream image" or "non-mainstream image." Non-mainstream images refer to portraits with features such as exaggerated hairstyles, brightly colored hair, or a disheveled appearance. Therefore, the first sample image set can be a collection of both mainstream and non-mainstream images (both carrying category labels). The second sample image set can be a collection of unlabeled portraits. Of course, the sample image set can include images from outside the target domain, such as non-human images (animals, landscapes, etc.). These non-target domain images can be referred to as the third sample image set. The third sample image set includes the third sample image and its category label. The third sample image is an image from outside the target domain corresponding to the target task. Therefore, the category label for the third sample image could be: "non-human image."

[0078] For example, in the sample image set, there is a large portion of unlabeled human portrait data, referred to as set U (the second sample image set). In the remaining labeled data, mainstream images (referred to as set Lz) constitute the majority, while non-mainstream images (referred to as set Lf) constitute a smaller portion. Sets Lz and Lf together form the first sample image set. In addition, there is a lot of online image data that does not belong to humans, such as objects, anime, landscapes, etc. This type of data, referred to as set N (the third sample image set), also needs to be prepared.

[0079] Step S320: Perform image conversion processing on the sample image set to obtain an auxiliary image set.

[0080] In specific implementation, the auxiliary image set includes a first augmented image set, which includes a first augmented image and its category label. The first augmented image is obtained by performing image enhancement processing on the first sample image, and its category label matches that of the first sample image. Computer equipment can perform image transformation processing on the sample image set to obtain the auxiliary image set. Image processing can include image enhancement processing and image segmentation processing. An auxiliary task refers to a task constructed to assist the target task. An auxiliary task is a task that the model automatically learns to automatically generate new labels (category labels or location labels) for images in the sample image set. For example, an auxiliary task could be: identifying the location of the left eye in a portrait or determining the feature similarity between images.

[0081] Specifically, image enhancement processing can include: 1. random cropping; 2. horizontal image flipping; 3. image rotation; 4. adding noise; 5. modifying image brightness; 6. modifying image saturation; 7. covering a small portion of image content; 8. Gaussian blur. Of course, the above image enhancement processing schemes can also be randomly combined, for example, rotating the image first and then adding Gaussian blur, etc. Image enhancement processing can also include various blurring methods (not limited to Gaussian blur, such as intelligent blur and fast blur); noise can be added using salt-and-pepper noise, motion blur, etc.; the image can also be covered with patterns of arbitrary shapes, etc.; or various filters can be used to stylize the image, such as calculating the image's gradient, etc.

[0082] Specifically, regarding image segmentation, "Shamate" images (non-mainstream images) have strong characteristics compared to images of ordinary people (mainstream images), namely, bizarre hairstyles, rich hair colors, and non-mainstream makeup (such as black lipstick, dark eyeshadow, earrings, nose rings, etc.). Therefore, these characteristics can be used to further augment the data.

[0083] like Figure 4a As shown, Figure 4a This is a flowchart illustrating an image segmentation process provided in an embodiment of this application. Each image in the human images (sets U, Lz, Lf) of the training images can be uniformly divided into three parts from top to bottom (each part is called an image patch). The corresponding location labels include: top part, middle part, and bottom part. Alternatively, besides uniformly dividing the image into three parts, it can be extended to four parts: dividing the image horizontally and vertically into four equal parts. The resulting four image patches would then have corresponding location labels: top left, bottom left, top right, and bottom right. Figure 4b As shown, Figure 4b This is a schematic diagram of another image segmentation process provided in an embodiment of this application. Figure 4b In the training image set (U, Lz, Lf), each human image is evenly divided into three parts from top to bottom. Alternatively, it can be extended to divide into nine parts, that is, the image is divided into three equal parts horizontally and vertically, resulting in a tic-tac-toe grid of nine blocks. The corresponding location labels include nine labels, each corresponding to a specific grid cell in a 3x3 grid.

[0084] Finally, all image patches obtained after image segmentation in each image set are labeled as U*, Lz*, and Lf*. Specifically, the set corresponding to all image patches obtained from all image segments in set U is U*, the set corresponding to all image patches obtained from all image segments in set Lz is Lz*, and the set corresponding to all image patches obtained from all image segments in set Lf is Lf*.

[0085] It should be noted that if the sample image set includes a third sample image set N, since set N contains a set of non-human images, the image data is large enough, and therefore the non-human image set N does not need to undergo image enhancement and image segmentation processing.

[0086] Step S330: Select the training dataset for the target task from the sample image set and the auxiliary image set, and train the first model using the training dataset for the target task.

[0087] In specific implementation, the training dataset for the target task includes at least a first set of sample images and a first set of augmented images. Additionally, the sample image set includes a third set of sample images, which comprises the third sample images and their category labels. These third sample images are images from the non-target domain corresponding to the target task. In summary, the training dataset for the target task includes: a first set of sample images, a first set of augmented images, and a third set of sample images. Furthermore, the first model includes a first feature extractor and a first classifier connected to the first feature extractor.

[0088] In one possible implementation, the computer device invokes a first model to identify a first sample image in a first sample image set, and obtains a category prediction label corresponding to the first sample image; the computer device invokes the first model to identify a first augmented image in a first augmented image set, and obtains a category prediction label corresponding to the first augmented image; the computer device invokes the first model to identify a third sample image in a third sample image set, and obtains a category prediction label corresponding to the third sample image; the computer device adjusts the model parameters of the first model based on the differences between the category label and the category prediction label of the first sample image, the differences between the category label and the category prediction label of the first augmented image, and the differences between the category label and the category prediction label of the third sample image.

[0089] Specifically, the category labels are: non-human images, mainstream images, and non-mainstream images. In the third sample image set, each image is labeled as a non-human image, while each image in the first sample image set is labeled as either a mainstream or non-mainstream image. Since image enhancement processing does not change the image labels, each image in the first augmented image set is also labeled as either a mainstream or non-mainstream image, and there is a one-to-one correspondence between the labels carried by each image in the first augmented image set and those carried by each image in the first sample image set. For example, if an image in the first sample image set is labeled as a mainstream image, then the corresponding image in the first augmented image set will also be labeled as a mainstream image. Of course, in addition to mainstream or non-mainstream images, the category labels for each image in the first sample image set can also include additional label types, such as "wedding" and "ancient costume." Because the images in these two categories ("wedding," "ancient costume") are visually similar to "Shamate" (non-mainstream images), they can serve as additional supervisory information, improving the accuracy of the classification network in classifying images. Furthermore, the added extra label types can also be used to identify wedding images and images in traditional costumes in online images, serving as additional output for the classification task and enhancing its value.

[0090] For example, please see Figure 4c , Figure 4c This is a schematic diagram illustrating a process for training a first model according to an embodiment of this application. It can be used as follows... Figure 4c The network shown is called Network A. This classification network A consists of a feature extractor G (the first feature extractor) plus a final fully connected layer (FC1). Network A learns image labels. Since it trains a classifier, the loss function used after processing by the fully connected layer FC1 (the first classifier) ​​can be a commonly used classification function, such as softmax, weighted cross entropy, large margin loss (center loss, cosface, arcface), etc. The feature extractor can be composed of any high-performance neural network structure, such as deep residual networks (ResNet), dense connection networks (DenseNet), etc. Of course, the feature extractor can also be a Visual Geometry Group Network (VGG), AlexNet, SEResNet, Resnext, etc. It should be noted that... Figure 4cThe network A shown belongs to the main task (target task) in model training.

[0091] Step S340: Select a training dataset for the auxiliary task of the target task from the sample image set and the auxiliary image set, and train the second model using the training dataset for the auxiliary task.

[0092] In specific implementation, the second model includes a first auxiliary model and a second auxiliary model. The first auxiliary model can be a location-based classification model, and the second auxiliary model can be a feature similarity-based discriminant model. The first auxiliary model includes a second feature extractor and a second classifier connected to it; the first and second feature extractors share the model structure and parameters. The second auxiliary model includes two third feature extractors and a similarity discriminator; the first and third feature extractors share the model structure and parameters. Therefore, the second model includes a second feature extractor and a second classifier connected to it, and two third feature extractors and similarity discriminators connected to each of the two third feature extractors; the first feature extractor shares the model structure and parameters with the second and third feature extractors.

[0093] Of course, the model parameters of the second feature extractor can also be used as new model parameters of the first feature extractor to retrain the first model, that is, the first model and the second model can be trained alternately.

[0094] Of course, since the trained second model is used to construct auxiliary tasks, the second model in this application embodiment can have a third auxiliary model, or even a fourth auxiliary model, in addition to the first and second auxiliary models. This application embodiment does not limit this. Appropriate auxiliary tasks can be constructed according to specific circumstances. It should be noted that the classification result corresponding to the first auxiliary model is the image location label, and the discrimination result corresponding to the second auxiliary model is the feature similarity between images.

[0095] Step S350: Determine the trained first model as the target model for performing the target task.

[0096] In practice, since the first model corresponds to the target task and the second model corresponds to the auxiliary task, when both the trained first and second models meet the model convergence condition, the target model is the trained first model corresponding to the target task. Of course, the target model is used to recognize images within a target domain. This target domain includes, but is not limited to, human portraits, medicine, animals, and nature.

[0097] In one possible implementation, if either the first model or the second model fails to meet the model convergence condition, then the first and second models need to be trained alternately until both the trained first and second models meet the model convergence condition. Then, the trained first model is used as the target model for performing the target task.

[0098] In this embodiment, a joint training process of the first model and the second model includes: ① Training the first model first and updating the model parameters of the first feature extractor; ② Simultaneously updating the model parameters of the second feature extractor based on the updated model parameters of the first feature extractor; ③ Training the second model and updating the model parameters of the second feature extractor; ④ Conversely, simultaneously updating the model parameters of the first feature extractor based on the updated model parameters of the second feature extractor. This completes one joint training session, which can be called a loop process. Repeating the above loop process allows for joint alternating training of the first and second models. Of course, the above joint training process can also involve training the second model first and then the first model; this application does not limit this training order. Regardless of which model is trained first, the feature extractor used during the training of the other model is the feature extractor of the previously trained model, so that the feature extractor can simultaneously learn the model performance required by both models. Once both the first model and the second model reach convergence, the training of the first model and the second model is complete.

[0099] In one possible implementation, the target model includes a trained first feature extractor and a trained first classifier. The computer device displays an image recognition interface, which includes an image recognition control and an image import control. When the image import control is triggered, the image to be processed in the target domain is acquired and displayed on the image recognition interface. When the image recognition control is triggered, the trained first feature extractor is invoked to extract features from the image to be processed, obtaining the image features corresponding to the image to be processed. The trained first classifier is also invoked to recognize the image features corresponding to the image to be processed, obtaining the image category corresponding to the image to be processed. The image category corresponding to the image to be processed is then displayed on the image recognition interface.

[0100] For example, the image processing method provided in this application can be applied to the fields of human portraits, medicine, animals, and nature, etc. The following detailed explanation will use the human portrait field as an example. Figure 2e As shown, firstly, as Figure 2eAs shown in the left-middle figure (10), the user can select an image to be processed from the album, and then click the "Send" button (image import control). The computer device will then acquire the image to be processed in the target area and display the image to be processed on the image recognition interface. Then, as shown in the figure... Figure 2e As shown in the middle right figure (20), when the user clicks the "Image Recognition" button (image recognition control) in the image recognition interface, the computer device can call the target model (the first model after training) to perform image recognition on the image to be processed. Specifically, the first feature extractor after training extracts features from the image to be processed to obtain the image features corresponding to the image to be processed, and calls the first classifier after training to recognize the image features corresponding to the image to be processed to obtain the image category corresponding to the image to be processed. Finally, the image category corresponding to the image to be processed (assuming it is a non-mainstream image) can be displayed in the image recognition interface.

[0101] Subsequently, post-processing can be performed on the image to be processed based on the image category it corresponds to. For example, the image to be processed can be intercepted, or the user who sent or uploaded the image to be processed can be warned or have their account banned, etc.

[0102] The image processing method provided in this application provides an image transformation process for a set of sample images used for model training, resulting in an auxiliary image set. Model training is then performed using both the sample image set and the auxiliary image set, thus solving the problem of training data collection, enriching the training data for model training, and facilitating the training of a more accurate model. Furthermore, during the training of the first model, a second model is selected to assist in training. This second model can share the model structure and parameters of the feature extractor with the first model. Training data is selectively chosen from the sample image set and the auxiliary image set based on the first and second models. This training data can include labeled or unlabeled images. Through this model structure and self-supervised / semi-supervised training methods, the trained first model can achieve better performance. Therefore, when using the trained first model to perform target task-oriented image recognition, superior image recognition results can be obtained, thereby improving the accuracy of image recognition.

[0103] Please see Figure 5 , Figure 5 This is a schematic flowchart of another image processing method provided in an embodiment of this application. The method is applied to a computer device, and... Figure 5 The specific implementation example is as follows: Figure 3 A specific embodiment corresponding to step S340 in the examples. For example... Figure 5 As shown, the data processing method may include steps S510 to S530. Wherein:

[0104] Step S510: Train the first auxiliary model using the training dataset of the auxiliary task.

[0105] In specific implementation, the first auxiliary model is a location-based classification model. The auxiliary image set includes a first segmented image set and a second segmented image set. The first segmented image set contains the first segmented image and its location labels; the first segmented image is obtained by segmenting the first sample image. The second segmented image set contains the second segmented image and its location labels; the second segmented image is obtained by segmenting the second image. The training dataset for the auxiliary task includes both the first and second segmented image sets. The location labels can specifically be: top, middle, bottom; or top left, top right, bottom left, bottom right, etc. Alternatively, the location labels can also be location coordinates, which can be x-coordinates and y-coordinates, or longitude and latitude coordinates, etc.

[0106] In one possible implementation, the computer device invokes a first auxiliary model to identify segmented images in the training dataset for the auxiliary task, obtaining the corresponding location prediction labels for the segmented images; the segmented images in the training dataset for the auxiliary task include a first segmented image and / or a second segmented image. The computer device adjusts the model parameters of the first auxiliary model based on the difference between the location labels of the segmented images and the location prediction labels.

[0107] For example, sets U*, Lz*, and Lf* all contain human images, and are each top, middle, and bottom patches of a complete image. Therefore, a classifier can be built to learn whether the input image is the top, middle, or bottom part of the image. Similarly, such a classifier consists of a feature extractor G (third feature extractor) and a fully connected layer FC2 (second discriminant unit model). See also... Figure 6a , Figure 6a This is a schematic diagram illustrating a process for training a second model according to an embodiment of this application. It can be used as follows... Figure 6a The network shown is called Network B. This design allows the feature extractor to learn the spatial features of images containing people. Hair features are generally located in the upper part of the image, while face features are located in the middle, and should be below hair features. During training, the feature extractor G from the previous step (Network A) is used. After training through the two parts (Network A and Network B), this feature extractor G can simultaneously learn the distinction between mainstream and non-mainstream features, as well as the spatial location information of images containing people. Since a classifier is being trained, the loss function used after FC2 can be a commonly used classification function, such as softmax, weighted cross entropy, etc.

[0108] Step S520: Adjust the model parameters of the third extractor in the second auxiliary model to the model parameters of the second feature extractor in the trained first auxiliary model.

[0109] In practical implementation, the second model shares the model structure and parameters of the feature extractor with the first model. Furthermore, the second model includes a first auxiliary model and a second auxiliary model. The first auxiliary model includes a second feature extractor and a second classifier connected to the second feature extractor. The second auxiliary model includes two third feature extractors and similarity discriminators connected to each of the two third feature extractors. Therefore, the second model shares the model structure and parameters of the feature extractor with the first model; specifically, the model parameters of the third feature extractors are adjusted to match the trained model parameters of the second feature extractor.

[0110] It should be noted that in this application, where the second model includes both a first auxiliary model and a second auxiliary model, the training is performed in the order of first training the first auxiliary model and then training the second auxiliary model. Therefore, after the computer device completes the training of the first auxiliary model, the model parameters of the third feature extractor in the second auxiliary model are kept consistent with the model parameters of the second feature extractor in the trained first auxiliary model.

[0111] If the second auxiliary model is trained first, followed by the first auxiliary model, then after training the second auxiliary model, the model parameters of the second feature extractor in the first auxiliary model are kept consistent with the model parameters of the third feature extractor in the trained second auxiliary model. In general, regardless of which auxiliary model is trained first, the model parameters of the untrained auxiliary model are kept consistent with the model parameters of the first-trained auxiliary model.

[0112] Step S530: Train the second auxiliary model using the training dataset of the auxiliary task.

[0113] In practice, the second auxiliary model is a discriminative model based on feature similarity. The auxiliary image set also includes a second augmented image set, which contains second augmented images obtained by enhancing the second sample images. The training dataset for the auxiliary task includes a sample image set and an auxiliary image set. Specifically, the training dataset for the auxiliary task includes: a sample image set, a first augmented image set, and a second augmented image set.

[0114] In one possible implementation, the computer device invokes a second auxiliary model to extract features from sample images in the sample image set, obtaining image features corresponding to the sample images. The computer device then invokes the second auxiliary model to extract features from augmented images in the auxiliary image set, obtaining image features corresponding to the augmented images. Based on the feature similarity between the image features corresponding to the sample images and the image features corresponding to the augmented images, the computer device trains the second auxiliary model.

[0115] For example, sets U', Lz', and Lf' are augmentations of image sets U, Lz, and Lf, so they should contain similar content. This allows us to utilize information from sets U and U', even if these sets do not contain supervisory labels indicating whether an image is non-mainstream. Here, the Siamese network result can be used as a metric to measure the similarity of the input image content. Please see [link to relevant documentation]. Figure 6b , Figure 6b This is a schematic diagram illustrating a process for training a second auxiliary model according to an embodiment of this application. Figure 6b As shown, Figure 6b The feature extractors G shown are two identical models, hence the name Siamese network. However, the input parameters of the models are different: the upper model has inputs U, Lz, Lf, and the lower model has inputs U', Lz', Lf'. We call this network network C. Network C can include two third feature extractors and similarity discriminators connected to the third feature extractors respectively. In this step, we still use the feature extractor G (third feature extractor) used in the previous two steps.

[0116] In one possible implementation, the computer device obtains the cosine distance between the image features corresponding to the sample image and the image features corresponding to the augmented image; the computer device determines the feature similarity between the sample image and the augmented image based on the cosine distance; the computer device determines the loss function of the second model based on the sum of feature similarities between all images in the sample image set and all images in the auxiliary image set; and the computer device adjusts the model parameters of the second auxiliary model based on the loss function.

[0117] For example, the input to a Siamese network is an image i from sets U, Lz, Lf, and its augmentation j (so image j comes from sets U', Lz', Lf'). If the class labels of image i and image j are the same, it means that j is an augmentation of i, and the two images are identical in content. Therefore, after passing through the feature extractor G, images i and j will yield two features G(i) and G(j). The final loss function measures the similarity between these two features: the more similar the features are, the smaller the penalty value of the loss function should be; conversely, the less similar the features are, the larger the penalty value should be. If j is not an augmentation of i, then the final loss function should make their features G(i) and G(j) as dissimilar as possible.

[0118] When training network C, n images (set B) are first randomly selected from sets U, Lz, and Lf. Image augmentation is then used to obtain one augmented image (set B') for each image. Therefore, the network input in one batch consists of 2n images. The original image i from sets U, Lz, and Lf is input into the first branch of the network, and the augmented image j is input into the second branch. After the two images pass through a feature extractor G with identical parameters, features G(i) and G(j) are obtained. We use cosine distance to measure the similarity between features. Of course, besides using cosine distance to calculate the similarity between features, Chebyshev distance, vector inner product, Hamming distance, edit distance, etc., can also be used to determine the similarity between features.

[0119] The cosine distance is shown in formula (1):

[0120] cos(i,j)=[G(i)·G(j)] / [||G(i)||*||G(j)||](1)

[0121] In formula (1), the value of cos(i,j) is between 0 and 1. The larger the value, the greater the similarity between graph i and graph j.

[0122] Furthermore, the similarity between features is shown in formula (2):

[0123] simi(i,j)=exp{cos(i,j)} / λ(2)

[0124] In formula (2), λ is a fixed parameter, which we can set as λ = 0.1.

[0125] Next, we define the loss function Loss(i,j) generated by the two graphs i and j as shown in formula (3):

[0126] Loss(i,j)=-log{simi(i,j) / [∑{k=1,2,...,n,k≠i'}simi(i,k)]}(3)

[0127] In formula (3), i' is the augmentation of image i. The denominator of the above formula represents the sum of the feature similarities between image i and all augmented images of i' that are not its augmentations. This means that if there are n original images, there are also n augmented images, and the denominator of the above formula is the sum of n-1 feature similarities simi.

[0128] Finally, we define the loss function for the entire batch as shown in Equation (4):

[0129] L=∑{i∈B}∑{j∈B'}Loss(i,j)(4)

[0130] Therefore, the final loss function takes into account the similarity between all original images B and augmented images B' in a batch.

[0131] In addition, the loss function for the entire batch can be modified as shown in formula (5):

[0132] L=∑{i∈B}∑{j∈B'}Loss(i,j)+Loss(j,i)(5)

[0133] Finally, after constructing the three networks (Network A, Network B, and Network C), we iteratively train each of the three steps until all networks converge. Specifically, we first train the classification network A from step 1; then, we feed the trained feature extractor G into step 2 to train the classification network B; next, we feed the feature extractor G from step 2 into step 3 to train the network C; and finally, we feed the feature extractor G from step 3 into step 1 to train the classification network A, and so on.

[0134] It should be noted that in this application, where the second model includes a first auxiliary model (Network A) and a second auxiliary model (Network B), the training is performed in the order of training Network A first, followed by training Network B. Therefore, after the computer device completes the training of Network A, the model parameters of the third feature extractor in Network B are kept consistent with the model parameters of the second feature extractor in the trained Network A.

[0135] If network B is trained first, and then network A is trained, the model parameters of the second feature extractor in network A are kept consistent with the model parameters of the third feature extractor in the trained network B after network B is completed. In general, regardless of which auxiliary model is trained first, the model parameters of the untrained auxiliary model are kept consistent with the model parameters of the other auxiliary model that was trained first.

[0136] In this scheme, the second model assists the first model in training. Furthermore, in this embodiment, the second model includes a first auxiliary model and a second auxiliary model. By alternately training the first and second auxiliary models, the second model is trained so that the final trained second model can simultaneously identify the location of an object in an image and recognize the feature similarity between images. Further, training the second model assists the training of the first model. Since the first and second models share model structure and parameters, the training of the second model updates the model structure and parameters of the feature extractor and shares them with the first model. This allows the trained feature extractor of the first model to extract features for both the target task and the auxiliary task, enabling the trained first model to perform the auxiliary task while simultaneously performing the target task, thus improving the capabilities of the first model.

[0137] In one possible implementation, the second model is a first auxiliary model, which is a location-based classification model. The second model can then be trained using the training dataset for the auxiliary task. This training dataset includes both a first set of segmented images and a second set of segmented images.

[0138] In specific implementation, the auxiliary image set includes a first segmented image set and a second segmented image set. The first segmented image set contains the first segmented image and its location labels. The first segmented image is obtained by segmenting the first sample image. The second segmented image set contains the second segmented image and its location labels. The second segmented image is obtained by segmenting the second image. The location labels can be: top, middle, bottom; or top left, top right, bottom left, bottom right, etc. Alternatively, the location labels can be location coordinates, which can be horizontal and vertical coordinates, or longitude and latitude coordinates, etc.

[0139] In one possible implementation, the computer device invokes a first auxiliary model to identify segmented images in the training dataset for the auxiliary task, obtaining the corresponding location prediction labels for the segmented images; the segmented images in the training dataset for the auxiliary task include a first segmented image and / or a second segmented image. The computer device adjusts the model parameters of the first auxiliary model based on the difference between the location labels of the segmented images and the location prediction labels.

[0140] In one possible implementation, a second model is trained using auxiliary image sets (a first segmented image set and a second segmented image set) until the second model reaches a convergence condition, at which point training of the second model stops. The convergence condition can be any of the following: the loss function of the second model is less than a set threshold; the loss function of the second model has stabilized and no longer changes as training continues; all training data used to train the second model (e.g., each image in the first and second segmented image sets) has been used in the training; the number of training iterations of the second model reaches a reference training quantity threshold (wherein this reference training quantity threshold is much smaller than the training threshold typically required for model training), etc.

[0141] Furthermore, after stopping training the second model, the model parameters of the first model can be adjusted to match those of the second model. The adjusted first model is then used as the new first model and trained. Next, the parameters of the second feature extractor in the second model are adjusted to match the parameters of the first feature extractor in the trained first model. This process is repeated, alternating between training the first and second models.

[0142] In this scheme, the number and structure of the second model are determined based on the actual auxiliary task. For example, in this embodiment, if the auxiliary task requires identifying the location of an object in an image, then the second model can select a first auxiliary model, which can be a location-based classification model. Furthermore, training the second model assists in training the first model. Since the first and second models share model structure and parameters, the training of the second model updates the model structure and parameters of the feature extractor and shares them with the first model. This allows the trained feature extractor of the first model to extract features for both the target task and the auxiliary task, enabling the trained first model to perform the auxiliary task while simultaneously executing the target task, thus enhancing the capabilities of the first model.

[0143] In one possible implementation, the second model is a second auxiliary model, which is a discriminative model based on feature similarity. This second model can be trained using the training dataset for the auxiliary task. The training dataset for the auxiliary task includes a set of sample images and a set of auxiliary images.

[0144] In a specific implementation, the auxiliary image set also includes a first augmented image set and a second augmented image set. The second augmented image set contains a second augmented image, which is obtained by performing image enhancement processing on the second sample image. Specifically, the auxiliary task can be to determine the feature similarity between images in the sample image set and images in the auxiliary image set.

[0145] In one possible implementation, the computer device invokes a second auxiliary model to extract features from sample images in the sample image set, obtaining image features corresponding to the sample images. The computer device then invokes the second auxiliary model to extract features from augmented images in the auxiliary image set, obtaining image features corresponding to the augmented images. Based on the feature similarity between the image features corresponding to the sample images and the image features corresponding to the augmented images, the computer device trains the second auxiliary model.

[0146] In one possible implementation, the computer device obtains the cosine distance between the image features corresponding to the sample image and the image features corresponding to the augmented image; the computer device determines the feature similarity between the sample image and the augmented image based on the cosine distance; the computer device determines the loss function of the second model based on the sum of feature similarities between all images in the sample image set and all images in the auxiliary image set; the computer device adjusts the model parameters of the second auxiliary model based on the loss function. Based on this, a new adjusted second auxiliary model is obtained, and the adjusted second auxiliary model continues to be trained on a training dataset including the sample image set and the auxiliary image set until the adjusted second auxiliary model satisfies the model convergence condition.

[0147] Similarly, the second model is trained using the sample image set and the auxiliary image set until it reaches the convergence condition, at which point training of the second model stops. The convergence condition can be any of the following: the loss function of the second model is less than a set threshold; the loss function of the second model has stabilized and no longer changes as training continues; all training data used to train the second model (e.g., each image in the first and second segmented image sets) has participated in the training; the number of training iterations of the second model reaches a reference training quantity threshold (wherein this reference training quantity threshold is much smaller than the training threshold that is typically required for model training), etc.

[0148] Furthermore, after stopping training the second model, the model parameters of the first model can be adjusted to match those of the second model. The adjusted first model is then used as the new first model and trained. Next, the parameters of the second feature extractor in the second model are adjusted to match the parameters of the first feature extractor in the trained first model. This process is repeated, alternating between training the first and second models.

[0149] The image processing method provided in this application constructs an auxiliary task to assist the target task. Training data is selectively chosen from the sample image set and the auxiliary image set based on the first model and the second model. This training data can include labeled or unlabeled images. Through this model structure and self-supervised / semi-supervised training, the trained first model can achieve better performance. When using the trained first model to perform the target task to recognize images in the target domain, superior image recognition results can be obtained, thereby improving the accuracy of image recognition. Furthermore, this solution uses a semi-supervised plus self-supervised approach, which can make great use of online collected data and does not require a large amount of manual annotation. Therefore, it can significantly shorten the iteration time of this task. For example, when the trained target model is applied to the portrait domain to recognize non-mainstream images, the target model can not only learn the feature differences between non-mainstream images (such as "Shamate" images) and mainstream images (such as images of normal people), but also learn the spatial distribution and / or feature similarity of non-mainstream images, thus achieving high accuracy and recall.

[0150] In this scheme, the number and structure of the second model are determined based on the actual auxiliary task. For example, in this embodiment, if the auxiliary task requires identifying feature similarity between images, then a second auxiliary model can be selected, which can be a discriminative model based on feature similarity. Furthermore, training the second model assists in training the first model. Since the first and second models share model structure and parameters, the training of the second model updates the model structure and parameters of the feature extractor and shares them with the first model. This allows the trained feature extractor of the first model to extract features for both the target task and the auxiliary task, enabling the trained first model to perform the auxiliary task while simultaneously executing the target task, thus enhancing the capabilities of the first model.

[0151] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Figure 7This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. This image processing apparatus can be applied to... Figures 3 to 6b The computer device in the corresponding method embodiment. The image processing apparatus may be a computer program (including program code) running on the computer device, for example, the image processing apparatus may be an application software; the apparatus may be used to perform the corresponding steps in the method provided in the embodiments of this application. The image processing apparatus may include:

[0152] The acquisition unit 710 is used to acquire a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and a category label for the first sample image. The second set of sample images includes a second sample image. Both the first sample image and the second sample image are images in the target domain corresponding to the target task.

[0153] The processing unit 720 is used to perform image conversion processing on the sample image set to obtain an auxiliary image set; the auxiliary image set includes a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained by performing image enhancement processing on the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image;

[0154] Training unit 730 is configured to select a training dataset for the target task from the sample image set and the auxiliary image set, and train a first model using the training dataset for the target task; the training dataset for the target task includes at least the first sample image set and the first augmented image set.

[0155] The training unit 730 is further configured to select a training dataset for an auxiliary task of the target task from the sample image set and the auxiliary image set, and train a second model using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model;

[0156] The determining unit 740 is used to determine the trained first model as the target model for performing the target task, wherein the target model is used to identify images in the target domain.

[0157] In one possible implementation, the sample image set further includes a third sample image set, which includes third sample images and category labels for the third sample images; the third sample images are images from a non-target domain corresponding to the target task; the training dataset for the target task further includes the third sample image set.

[0158] Training unit 730 trains a first model using the training dataset of the target task, including:

[0159] The first model is invoked to identify the first sample image in the first sample image set, and the category prediction label corresponding to the first sample image is obtained.

[0160] The first model is invoked to identify the first augmented image in the first augmented image set, and the category prediction label corresponding to the first augmented image is obtained;

[0161] The first model is invoked to identify the third sample image in the third sample image set, and the category prediction label corresponding to the third sample image is obtained.

[0162] The model parameters of the first model are adjusted based on the differences between the category label of the first sample image and the predicted category label of the first sample image, the differences between the category label of the first augmented image and the predicted category label of the first augmented image, and the differences between the category label of the third sample image and the predicted category label of the third sample image.

[0163] In one possible implementation, the auxiliary image set further includes a first segmented image set and a second segmented image set. The first segmented image set contains a first segmented image and its location labels. The first segmented image is obtained by segmenting the first sample image. The second segmented image set contains a second segmented image and its location labels. The second segmented image is obtained by segmenting the second image. The training dataset for the auxiliary task includes the first segmented image set and the second segmented image set. The second model is a first auxiliary model.

[0164] Training unit 730 trains a second model using the training dataset of the auxiliary task, including:

[0165] The second model is invoked to identify the segmented images in the training dataset of the auxiliary task, and the location prediction labels corresponding to the segmented images are obtained; the segmented images in the training dataset of the auxiliary task include a first segmented image and / or a second segmented image;

[0166] The model parameters of the second model are adjusted based on the difference between the location labels and the predicted location labels of the segmented image.

[0167] In one possible implementation, the auxiliary image set further includes a second augmented image set, which contains a second augmented image obtained by performing image enhancement processing on the second sample image; the training dataset for the auxiliary task includes the sample image set and the auxiliary image set; the second model is a second auxiliary model.

[0168] Training unit 730 trains a second model using the training dataset of the auxiliary task, including:

[0169] The second model is invoked to extract features from the sample images in the sample image set to obtain the image features corresponding to the sample images.

[0170] The second model is invoked to extract features from the augmented images in the auxiliary image set to obtain the image features corresponding to the augmented images;

[0171] The second model is trained based on the feature similarity between the image features corresponding to the sample image and the image features corresponding to the augmented image.

[0172] In one possible implementation, the second model includes a first auxiliary model and a second auxiliary model; the first auxiliary model is a location-based classification model; and the second auxiliary model is a feature-based discrimination model.

[0173] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor;

[0174] The second model is the first auxiliary model, which includes a second feature extractor and a second classifier connected to the second feature extractor; the first feature extractor and the second feature extractor share the model structure and model parameters.

[0175] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor;

[0176] The second model is a second auxiliary model, which includes two third feature extractors and a similarity discriminator; the first feature extractor and the third feature extractors share the model structure and model parameters.

[0177] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor;

[0178] The second model includes a first auxiliary model and a second auxiliary model. The first auxiliary model includes a second feature extractor and a second classifier connected to the second feature extractor.

[0179] The second auxiliary model includes two third feature extractors and similarity discriminators connected to the two third feature extractors respectively; the first feature extractor shares the model structure and model parameters with the second feature extractor and the third feature extractors.

[0180] In one possible implementation, the target task includes identifying image categories; the processing unit 720 is further configured to perform the following operations:

[0181] The image recognition interface is displayed, which includes an image recognition control and an image import control.

[0182] When the image import control is triggered, the image to be processed in the target area is acquired and displayed on the image recognition interface.

[0183] When the image recognition control is triggered, the trained target model is invoked to extract features from the image to be processed, thereby obtaining the image features corresponding to the image to be processed. The trained target model is also invoked to identify the image features corresponding to the image to be processed, thereby obtaining the image category corresponding to the image to be processed.

[0184] The image category is displayed on the image recognition interface.

[0185] The image processing apparatus provided in this application can perform image transformation processing on the sample image set used for model training to obtain an auxiliary image set. Model training is then performed using both the sample image set and the auxiliary image set, thus solving the problem of training data collection, enriching the training data for model training, and facilitating the training of a more accurate model. Furthermore, during the training of the first model, a second model is selected to assist in training. This second model can share the model structure and model parameters of the feature extractor with the first model. Training data is selectively chosen from the sample image set and the auxiliary image set based on the first and second models. This training data can include labeled or unlabeled images. Through this model structure and self-supervised / semi-supervised training methods, the trained first model can achieve better performance. Therefore, when using the trained first model to perform target task-oriented image recognition in the target domain, superior image recognition results can be obtained, thereby improving the accuracy of image recognition.

[0186] Please see Figure 8 Please see Figure 8 , Figure 8This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The above... Figures 3 to 6b The computer device in the corresponding embodiment can be computer device 800, such as... Figure 7 As shown, computer device 800 may include a user interface 802, a processor 804, an encoder 806, and a memory 808. A signal receiver 816 is used to receive or transmit data via a cellular interface 810, a Wi-Fi interface 812, ..., or an NFC interface 814. The encoder 806 encodes the received data into a data format that can be processed by a computer. The memory 808 stores a computer program, and the processor 804 is configured to perform the steps in any of the above method embodiments via the computer program. The memory 808 may include volatile memory (e.g., dynamic random access memory DRAM) and may also include non-volatile memory (e.g., one-time programmable read-only memory OTPROM). In some instances, the memory 808 may further include memory remotely located relative to the processor 804, which can be connected to the computer device 800 via a network. The user interface 802 may include a keyboard 818 and a display 820.

[0187] exist Figure 8 In the computer device 800 shown, the processor 804 can be used to call computer programs stored in the memory 808 to achieve:

[0188] Obtain a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and a category label for the first sample image. The second set of sample images includes a second sample image. Both the first sample image and the second sample image are images from the target domain corresponding to the target task.

[0189] The sample image set is subjected to image transformation processing to obtain an auxiliary image set; the auxiliary image set includes a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained by performing image enhancement processing on the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image;

[0190] The training dataset for the target task is selected from the sample image set and the auxiliary image set, and the first model is trained using the training dataset for the target task; the training dataset for the target task includes at least the first sample image set and the first augmented image set.

[0191] A training dataset for an auxiliary task of the target task is selected from the sample image set and the auxiliary image set, and a second model is trained using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model;

[0192] The trained first model is determined as the target model for performing the target task, and the target model is used to identify images in the target domain.

[0193] In one possible implementation, the sample image set further includes a third sample image set, which includes third sample images and category labels for the third sample images; the third sample images are images from a non-target domain corresponding to the target task; the training dataset for the target task further includes the third sample image set.

[0194] Processor 804 trains a first model using the training dataset for the target task, including:

[0195] The first model is invoked to identify the first sample image in the first sample image set, and the category prediction label corresponding to the first sample image is obtained.

[0196] The first model is invoked to identify the first augmented image in the first augmented image set, and the category prediction label corresponding to the first augmented image is obtained;

[0197] The first model is invoked to identify the third sample image in the third sample image set, and the category prediction label corresponding to the third sample image is obtained.

[0198] The model parameters of the first model are adjusted based on the differences between the category label of the first sample image and the predicted category label of the first sample image, the differences between the category label of the first augmented image and the predicted category label of the first augmented image, and the differences between the category label of the third sample image and the predicted category label of the third sample image.

[0199] In one possible implementation, the auxiliary image set further includes a first segmented image set and a second segmented image set. The first segmented image set contains a first segmented image and its location labels. The first segmented image is obtained by segmenting the first sample image. The second segmented image set contains a second segmented image and its location labels. The second segmented image is obtained by segmenting the second image. The training dataset for the auxiliary task includes the first segmented image set and the second segmented image set. The second model is a first auxiliary model.

[0200] Processor 804 trains a second model using the training dataset of the auxiliary task, including:

[0201] The second model is invoked to identify the segmented images in the training dataset of the auxiliary task, and the location prediction labels corresponding to the segmented images are obtained; the segmented images in the training dataset of the auxiliary task include a first segmented image and / or a second segmented image;

[0202] The model parameters of the second model are adjusted based on the difference between the location labels and the predicted location labels of the segmented image.

[0203] In one possible implementation, the auxiliary image set further includes a second augmented image set, which contains a second augmented image obtained by performing image enhancement processing on the second sample image; the training dataset for the auxiliary task includes the sample image set and the auxiliary image set; the second model is a second auxiliary model.

[0204] Processor 804 trains a second model using the training dataset of the auxiliary task, including:

[0205] The second model is invoked to extract features from the sample images in the sample image set to obtain the image features corresponding to the sample images.

[0206] The second model is invoked to extract features from the augmented images in the auxiliary image set to obtain the image features corresponding to the augmented images;

[0207] The second model is trained based on the feature similarity between the image features corresponding to the sample image and the image features corresponding to the augmented image.

[0208] In one possible implementation, the second model includes a first auxiliary model and a second auxiliary model; the first auxiliary model is a location-based classification model; and the second auxiliary model is a feature-based discrimination model.

[0209] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor;

[0210] The second model is the first auxiliary model, which includes a second feature extractor and a second classifier connected to the second feature extractor; the first feature extractor and the second feature extractor share the model structure and model parameters.

[0211] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor;

[0212] The second model is a second auxiliary model, which includes two third feature extractors and a similarity discriminator; the first feature extractor and the third feature extractors share the model structure and model parameters.

[0213] In one possible implementation, the first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor.

[0214] The second model includes a first auxiliary model and a second auxiliary model. The second model includes a second feature extractor and a second classifier connected to the second feature extractor, and includes two third feature extractors and similarity discriminators connected to the two third feature extractors respectively. The first feature extractor shares the model structure and model parameters with the second feature extractor and the third feature extractors.

[0215] In one possible implementation, the target task includes identifying image categories; the processor 804 is also configured to perform the following operations:

[0216] The image recognition interface is displayed, which includes an image recognition control and an image import control.

[0217] When the image import control is triggered, the image to be processed in the target area is acquired and displayed on the image recognition interface.

[0218] When the image recognition control is triggered, the trained target model is invoked to extract features from the image to be processed, thereby obtaining the image features corresponding to the image to be processed. The trained target model is also invoked to identify the image features corresponding to the image to be processed, thereby obtaining the image category corresponding to the image to be processed.

[0219] The image category is displayed on the image recognition interface.

[0220] It should be understood that the computer device 800 described in the embodiments of this application can execute the foregoing text. Figures 3 to 6b The description of the test analysis method in the corresponding embodiments can also be performed as described above. Figure 7 The description of the test and analysis apparatus in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0222] Furthermore, it should be noted that this application embodiment also provides a computer storage medium, which stores a computer program executed by the aforementioned test and analysis device. This computer program includes program instructions, and when the processor executes these program instructions, it can execute the aforementioned... Figures 3 to 6b The methods described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated. For technical details not disclosed in the computer storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0223] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figures 3 to 6b The methods described in the corresponding embodiments are therefore not repeated here.

[0224] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0225] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of the invention.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and a category label for the first sample image. The second set of sample images includes a second sample image. Both the first sample image and the second sample image are images from the target domain corresponding to the target task. The sample image set is subjected to image transformation processing to obtain an auxiliary image set; the auxiliary image set includes at least a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained by performing image enhancement processing on the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image; A training dataset for the target task is selected from the sample image set and the auxiliary image set, and a first model is trained using the training dataset for the target task; the training dataset for the target task includes at least the first sample image set and the first augmented image set; wherein, the first model is a classification model based on image features, and the first model includes a first feature extractor and a first classifier connected to the first feature extractor; A training dataset for an auxiliary task of the target task is selected from the sample image set and the auxiliary image set, and a second model is trained using the training dataset for the auxiliary task. The second model shares the model structure and model parameters of the feature extractor with the first model. The second model is a model that assists the first model in training, and the number and structure of the second model are determined based on the auxiliary task. The second model includes a first auxiliary model and a second auxiliary model. The first auxiliary model is a location-based classification model, including a second feature extractor and a second classifier connected to the second feature extractor. The second auxiliary model is a feature similarity-based discriminant model, including two third feature extractors and similarity discriminators connected to the two third feature extractors respectively. The first feature extractor shares the model structure and model parameters with the second feature extractor and the third feature extractors. The first model and the second model are jointly trained until both the trained first model and the trained second model meet the model convergence condition. The trained first model is then determined as the target model for performing the target task. The target model is used to recognize images in the target domain. The target model can also perform the auxiliary task while performing the target task.

2. The method according to claim 1, characterized in that, The sample image set also includes a third sample image set, which includes third sample images and category labels for the third sample images; the third sample images are images from non-target domains corresponding to the target task; the training dataset for the target task also includes the third sample image set. Training the first model using the training dataset for the target task includes: The first model is invoked to identify the first sample image in the first sample image set, and the category prediction label corresponding to the first sample image is obtained. The first model is invoked to identify the first augmented image in the first augmented image set, and the category prediction label corresponding to the first augmented image is obtained; The first model is invoked to identify the third sample image in the third sample image set, and the category prediction label corresponding to the third sample image is obtained. The model parameters of the first model are adjusted based on the differences between the category label of the first sample image and the predicted category label of the first sample image, the differences between the category label of the first augmented image and the predicted category label of the first augmented image, and the differences between the category label of the third sample image and the predicted category label of the third sample image.

3. The method according to claim 1, characterized in that, The auxiliary image set further includes a first segmented image set and a second segmented image set. The first segmented image set contains a first segmented image and its location labels. The first segmented image is obtained by segmenting the first sample image. The second segmented image set contains a second segmented image and its location labels. The second segmented image is obtained by segmenting the second sample image. The training dataset for the auxiliary task includes the first segmented image set and the second segmented image set. The training of the second model using the training dataset of the auxiliary task includes: The second model is invoked to identify the segmented images in the training dataset of the auxiliary task, and the location prediction labels corresponding to the segmented images are obtained; the segmented images in the training dataset of the auxiliary task include a first segmented image and / or a second segmented image; The model parameters of the second model are adjusted based on the difference between the location labels and the predicted location labels of the segmented image.

4. The method according to claim 1, characterized in that, The auxiliary image set also includes a second augmented image set, which contains a second augmented image, which is obtained by performing image enhancement processing on the second sample image; The training dataset for the auxiliary task includes the set of sample images and the set of auxiliary images; The training of the second model using the training dataset of the auxiliary task includes: The second model is invoked to extract features from the sample images in the sample image set to obtain the image features corresponding to the sample images. The second model is invoked to extract features from the augmented images in the auxiliary image set to obtain the image features corresponding to the augmented images; The second model is trained based on the feature similarity between the image features corresponding to the sample image and the image features corresponding to the augmented image.

5. The method according to any one of claims 1-3, characterized in that, The first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor; The second model is a first auxiliary model, which includes a second feature extractor and a second classifier connected to the second feature extractor; the first feature extractor and the second feature extractor share the model structure and model parameters.

6. The method according to claim 1, 2 or 4, characterized in that, The first model is a classification model based on image features; the first model includes a first feature extractor and a first classifier connected to the first feature extractor; The second model is a second auxiliary model, which includes two third feature extractors and a similarity discriminator; the first feature extractor and the third feature extractors share the model structure and model parameters.

7. The method according to claim 1, characterized in that, The target task includes identifying image categories; the method further includes: The image recognition interface is displayed, which includes an image recognition control and an image import control. When the image import control is triggered, the image to be processed in the target area is acquired and displayed on the image recognition interface. When the image recognition control is triggered, the trained target model is invoked to extract features from the image to be processed, thereby obtaining the image features corresponding to the image to be processed. The trained target model is also invoked to identify the image features corresponding to the image to be processed, thereby obtaining the image category corresponding to the image to be processed. The image category is displayed on the image recognition interface.

8. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire a set of sample images for model training. The set of sample images includes a first set of sample images and a second set of sample images. The first set of sample images includes a first sample image and a category label for the first sample image. The second set of sample images includes a second sample image. Both the first sample image and the second sample image are images from the target domain corresponding to the target task. A processing unit is configured to perform image conversion processing on the sample image set to obtain an auxiliary image set; the auxiliary image set includes at least a first augmented image set, the first augmented image set includes a first augmented image and a category label of the first augmented image, the first augmented image is obtained by performing image enhancement processing on the first sample image, and the category label of the first augmented image is consistent with the category label of the first sample image; A training unit is configured to select a training dataset for the target task from the sample image set and the auxiliary image set, and train a first model using the training dataset for the target task; the training dataset for the target task includes at least the first sample image set and the first augmented image set; wherein, the first model is a classification model based on image features, and the first model includes a first feature extractor and a first classifier connected to the first feature extractor; The training unit is further configured to select a training dataset for an auxiliary task of the target task from the sample image set and the auxiliary image set, and train a second model using the training dataset for the auxiliary task; the second model shares the model structure and model parameters of the feature extractor with the first model; the second model is a model that assists the first model in training, and the number and model structure of the second model are determined based on the auxiliary task; wherein, the second model includes a first auxiliary model and a second auxiliary model, the first auxiliary model is a location-based classification model, the first auxiliary model includes a second feature extractor and a second classifier connected to the second feature extractor; the second auxiliary model is a feature similarity-based discriminant model, the second auxiliary model includes two third feature extractors and similarity discriminators respectively connected to the two third feature extractors; the first feature extractor shares the model structure and model parameters with the second feature extractor and the third feature extractor; A determining unit is used to jointly train the first model and the second model until both the trained first model and the trained second model meet the model convergence condition. Then, the trained first model is determined as the target model for performing the target task. The target model is used to recognize images in the target domain. The target model can also perform the auxiliary task while performing the target task.

9. A computer device, characterized in that, include: Memory and processor; The memory contains computer programs; The processor is configured to load the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the method as described in any one of claims 1-7.

Images