Image recognition method, device, equipment and storage medium

Abstract

The application discloses an image recognition method and device, equipment and a storage medium, and belongs to the technical field of computers. Through the technical scheme provided by the embodiment of the application, under the premise that there are at least two categories of sample images in the sample data set, different ways can be used to sample in the sample data set to obtain first sample images, second sample images and third sample images. The first sample images are uniformly sampled sample images. The second sample images are sample images obtained based on at least two categories of first sampling probabilities, and the sampling mode indicated by the first sampling probability is that the probability of sampling the smaller number of categories is higher. The third sample images are sample images obtained after setting the same sampling probability for each category. The third sample images are used to train the image recognition model, and the obtained image recognition model has relatively accurate image recognition capability for different categories.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image recognition method, apparatus, device, and storage medium. Background Art

[0002] With the development of computer technology, artificial intelligence technology has flourished. As a branch of artificial intelligence technology, image recognition technology has an increasingly wide range of applications. For example, image recognition technology can be applied in face recognition scenarios. By using image recognition models to identify images containing faces, identity information corresponding to the faces can be obtained. Or it can be applied in medical scenarios. By using image recognition models to identify medical images, some pathological phenomena that are difficult for the human eye to distinguish can be discovered, thereby assisting doctors in determining treatment plans.

[0003] In related technologies, a large number of sample images are often required to train image recognition models. These sample images correspond to different types and are used to train the image recognition model's ability to recognize different types of images. Because the number of sample images of different types may be uneven, there may be a large number of sample images of one type and a small number of sample images of another type. For example, in the same sample dataset, there may be 1,000 sample images of type A and only 10 sample images of type B. In this case, the recognition results of the trained image recognition model will be biased towards the type of sample images with a larger number of samples, resulting in low image recognition accuracy. Summary of the Invention

[0004] The embodiments of the present application provide an image recognition method, apparatus, device, and storage medium that can improve the image recognition accuracy of an image recognition model. The technical solution is as follows:

[0005] In one aspect, an image recognition method is provided, the method comprising:

[0006] Uniformly sampling a sample data set to obtain a first sample image, wherein the sample data set includes sample images of at least two categories;

[0007] Based on the first sampling probabilities of the at least two categories, sampling sample images of the first category in the sample data set to obtain second sample images, wherein the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set;

[0008] Based on the second sampling probability, sampling the sample images of the second category in the sample data set to obtain a third sample image, wherein the at least two categories correspond to the same second sampling probability;

[0009] An image recognition model is trained based on the first sample image, the second sample image, the third sample image, and reference labels of the respective sample images, where the reference labels are used to represent categories of the corresponding sample images.

[0010] In one aspect, an image recognition method is provided, the method comprising:

[0011] Obtaining an image to be recognized;

[0012] Calling an image recognition model to recognize the image and obtain a predicted label for the image;

[0013] The image recognition model is trained based on the first sample image, the second sample image, the third sample image, and reference labels of the respective sample images, and the reference labels are used to indicate the category of the corresponding sample image;

[0014] The first sample image is obtained by uniformly sampling a plurality of sample images in a sample data set, the sample data set including sample images of at least two categories; the second sample image is obtained by sampling sample images of a first category in the sample data set based on a first sampling probability of the at least two categories; the third sample image is obtained by sampling sample images of a second category in the sample data set based on a second sampling probability;

[0015] The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and the at least two categories correspond to the same second sampling probability.

[0016] In one possible implementation, performing label prediction on the image through each branch network to obtain a probability distribution column of the image includes:

[0017] Performing feature extraction on the image through the convolution layer of the first branch network to obtain a first image feature of the image; and obtaining a first probability distribution column of the image based on the first image feature;

[0018] Performing feature extraction on the image through a second branch network to obtain a second image feature of the image; and obtaining a second probability distribution column of the image based on the second image feature;

[0019] Performing feature extraction on the image through a third branch network to obtain a third image feature of the image; and obtaining a third probability distribution column of the image based on the third image feature;

[0020] The first probability distribution column, the second probability distribution column, and the third probability distribution column are added together to obtain a probability distribution column of the image.

[0021] In one aspect, an image recognition device is provided, comprising:

[0022] a first sampling module, configured to uniformly sample a sample data set to obtain a first sample image, wherein the sample data set includes sample images of at least two categories;

[0023] a second sampling module, configured to sample sample images of a first category in the sample data set based on first sampling probabilities of the at least two categories to obtain second sample images, wherein the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set;

[0024] a third sampling module, configured to sample sample images of the second category in the sample data set based on a second sampling probability to obtain a third sample image, wherein the at least two categories correspond to the same second sampling probability;

[0025] A training module is used to train an image recognition model based on the first sample image, the second sample image, the third sample image and reference labels of each sample image, where the reference labels are used to represent the categories of the corresponding sample images.

[0026] In one possible implementation, the image recognition model includes multiple branch networks, each branch network has the same network structure, and the training module is used to perform label prediction on each sample image based on the each branch network to obtain the predicted label of each sample image; based on the first difference information between the predicted label of each sample image and the corresponding reference label, the image recognition model is trained.

[0027] In one possible implementation, the training module is configured to fuse first difference information between the predicted labels of the respective sample images and the corresponding reference labels to obtain a first loss value;

[0028] The image recognition model is trained based on the first loss value.

[0029] In one possible implementation, the training module is used to use a first weight, a second weight, and a third weight to weightedly sum the first difference information between the predicted labels of each sample image and the corresponding reference labels to obtain the first loss value. The first weight is negatively correlated with the ratio of the number of iterative cycles of the image recognition model to the total number of iterative cycles. The sum of the second weight and the first weight is the target value, and the third weight is greater than the first weight.

[0030] In one possible implementation, the training module is configured to perform feature extraction on the first sample image through a first branch network to obtain features of the first sample image; and obtain a predicted label of the first sample image based on the features of the first sample image;

[0031] Performing feature extraction on the second sample image through the second branch network to obtain features of the second sample image; and obtaining a predicted label of the second sample image based on the features of the second sample image;

[0032] Perform feature extraction on the third sample image through the third branch network to obtain features of the third sample image; and obtain a predicted label of the third sample image based on the features of the third sample image.

[0033] In one possible implementation, the apparatus further includes:

[0034] a predicted ranking information acquisition module, configured to acquire predicted ranking information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, wherein the predicted ranking information is used to represent the correlation between different sample images;

[0035] A supervised sorting information acquisition module, configured to acquire supervised sorting information based on the reference labels of the respective sample images;

[0036] The training module is used to train the image recognition model based on the first difference information and the second difference information, where the second difference information is the difference information between the predicted ranking information and the supervised ranking information.

[0037] In a possible implementation, the prediction ranking information acquisition module is configured to splice features of the first sample image, features of the second sample image, and features of the third sample image to obtain spliced ​​features;

[0038] Fully connected and normalized processing is performed on the splicing features to obtain the predicted ranking information.

[0039] In a possible implementation, the supervised ranking information acquisition module is configured to divide the reference label of the first sample image by the fusion label to obtain a first ranking label;

[0040] Dividing the reference label of the second sample image by the fusion label to obtain a second ranking label;

[0041] Dividing the reference label of the third sample image by the fusion label to obtain a third ranking label;

[0042] Combining the first sorting label, the second sorting label, and the third sorting label to obtain the supervised sorting information;

[0043] The fusion label is a label obtained by adding the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

[0044] In one possible implementation, the training module is configured to obtain a first loss value based on the first difference information;

[0045] Obtaining a second loss value based on the second difference information;

[0046] Performing a weighted summation on the first loss value and the second loss value to obtain a joint loss value;

[0047] The image recognition model is trained based on the joint loss value.

[0048] In one aspect, an image recognition device is provided, comprising:

[0049] An image acquisition module, used to acquire an image to be identified;

[0050] An image recognition module is used to call an image recognition model to recognize the image and obtain a predicted label for the image;

[0051] The image recognition model is trained based on the first sample image, the second sample image, the third sample image, and reference labels of the respective sample images, and the reference labels are used to indicate the category of the corresponding sample image;

[0052] The first sample image is obtained by uniformly sampling a plurality of sample images in a sample data set, the sample data set including sample images of at least two categories; the second sample image is obtained by sampling sample images of a first category in the sample data set based on a first sampling probability of the at least two categories; the third sample image is obtained by sampling sample images of a second category in the sample data set based on a second sampling probability;

[0053] The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and the at least two categories correspond to the same second sampling probability.

[0054] In one possible implementation, the image recognition model includes multiple branch networks, each branch network having the same network structure, and the image recognition module is used to perform label prediction on the image through each branch network to obtain a probability distribution column for the image, where the values ​​in the probability distribution column are used to represent the probability that the image corresponds to a label;

[0055] In response to a probability represented by any numerical value in the probability distribution column meeting a target probability condition, a label corresponding to the any numerical value is determined as the predicted label.

[0056] In one possible implementation, the image recognition module is configured to perform feature extraction on the image through a convolutional layer of a first branch network to obtain a first image feature of the image; and obtain a first probability distribution column of the image based on the first image feature;

[0057] Performing feature extraction on the image through a second branch network to obtain a second image feature of the image; and obtaining a second probability distribution column of the image based on the second image feature;

[0058] Performing feature extraction on the image through a third branch network to obtain a third image feature of the image; and obtaining a third probability distribution column of the image based on the third image feature;

[0059] The first probability distribution column, the second probability distribution column, and the third probability distribution column are added together to obtain a probability distribution column of the image.

[0060] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the image recognition method.

[0061] In one aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the image recognition method.

[0062] On the one hand, a computer program product or computer program is provided, which includes a program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned image recognition method.

[0063] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. Using the sample images obtained by the three sampling methods to train the image recognition model can improve the image recognition accuracy of the image model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 is a schematic diagram of an implementation environment of an image recognition method provided in an embodiment of the present application;

[0066] Figure 2 This is a schematic diagram of the structure of an image recognition model provided in an embodiment of the present application;

[0067] Figure 3 This is a flowchart of an image recognition method provided by an embodiment of the present application;

[0068] Figure 4 This is a flowchart of an image recognition method provided by an embodiment of the present application;

[0069] Figure 5 is a schematic diagram of a fracture image provided in an embodiment of the present application;

[0070] Figure 6 This is a flowchart of an image recognition method provided by an embodiment of the present application;

[0071] Figure 7 This is a schematic diagram of an interface provided by an embodiment of the present application;

[0072] Figure 8 1 is a schematic structural diagram of an image recognition device provided in an embodiment of the present application;

[0073] Figure 9 1 is a schematic structural diagram of an image recognition device provided in an embodiment of the present application;

[0074] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0075] Figure 11 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] To make the objectives, technical solutions, and advantages of this application more clear, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0077] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0078] The technical solutions provided in the embodiments of this application involve artificial intelligence (AI) technology. AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning. The embodiments of this application address reinforcement learning and computer vision within AI technology.

[0079] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies. The embodiments of this application relate to reinforcement learning technology in machine learning. Reinforcement learning is a branch of machine learning that is used to describe and solve the problem of an agent maximizing rewards or achieving specific goals through learning strategies during its interaction with the environment.

[0080] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and further processing the images to make them more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition. The embodiments of this application relate to image semantic understanding technology within computer vision technology.

[0081] The technical solutions provided in the embodiments of this application can also be combined with cloud technology. For example, the trained image segmentation model can be deployed on a cloud server. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. Cloud technology can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a critical support. Backend services for technical network systems, such as video websites, image websites, and more portals, require a large amount of computing and storage resources. With the rapid development and application of the internet industry, every item in the future may have its own unique identification mark, which will need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. Data from various industries requires a strong system backend, which can only be achieved through cloud computing.

[0082] Among them, the medical cloud in cloud technology refers to the use of "cloud computing" to create a medical and health service cloud platform based on new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents' medical treatment. For example, the current hospital appointment registration, electronic medical records, medical insurance, etc. are all products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and global layout. For example, the image segmentation model provided in the embodiment of the present application is deployed on the medical and health service cloud platform.

[0083] Normalization: Mapping sequences of numbers with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values ​​can be directly implemented as probabilities.

[0084] Learning Rate: This controls the model's learning progress. It guides how the model uses the gradient of the loss function to adjust network weights during gradient descent. If the learning rate is too high, the loss function may overshoot the global minimum, resulting in excessive loss. If the learning rate is too low, the loss function changes very slowly, significantly increasing the network's convergence complexity and making it more likely to become trapped in a local minimum or saddle point.

[0085] Epoch of model training: also known as the cycle of model training. In the embodiment of the present application, all the training data in the training dataset are used to train the image segmentation model, which is called one generation of training, i.e., one epoch.

[0086] Iterations of model training: In this embodiment, training a model using a batch of training data from a training dataset is called one round of training, or one iteration. A generation of model training includes at least one round of model training. For example, if a training dataset includes 100 training data points and a batch includes 20 training data points, the computer device will perform one round of model training based on the 20 training data points. After five rounds of model training, the computer device will have traversed all the training data in the training dataset, completing one generation of model training.

[0087] Optionally, the computer device provided in the embodiment of the present application can be implemented as a terminal or a server. The implementation environment composed of the terminal and the server is introduced below.

[0088] Figure 1 This is a schematic diagram of an implementation environment of an image recognition method provided in an embodiment of the present application, see Figure 1 , the implementation environment may include a terminal 110 and a server 140.

[0089] The terminal 110 is connected to the server 140 via a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 110 has an application installed and running that supports image recognition.

[0090] Optionally, server 140 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, a content delivery network (CDN), and big data and artificial intelligence platforms. In some embodiments, a user can upload an image to be recognized to server 140 via terminal 110, and server 140 will perform the image recognition method provided in the embodiments of the present application on the image uploaded by the user. After the recognition is completed, server 140 sends the recognition result to terminal 110, and the user can view the recognition result of the image through terminal 110.

[0091] Optionally, the terminal 110 generally refers to one of multiple terminals, and the embodiment of the present application only takes the terminal 110 as an example.

[0092] Those skilled in the art will appreciate that the number of terminals 110 may be greater or lesser. For example, there may be only one terminal 110, or there may be dozens, hundreds, or even more terminals 110. In this case, the implementation environment may also include other terminals. The present embodiment does not limit the number or device type of terminals.

[0093] After introducing the implementation environment of the image recognition method provided in the embodiment of the present application, the application scenario of the image recognition method provided in the embodiment of the present application is described below. It should be noted that the computer device in the following description can be implemented as the terminal 110 or the server 140 in the above implementation environment, and the embodiment of the present application does not limit this.

[0094] The image recognition method provided in the embodiments of the present application can be applied in a variety of image recognition scenarios. For example, it can be applied in scenarios of medical image recognition, building image recognition, vehicle image recognition, or material image recognition, etc., and the embodiments of the present application do not limit this.

[0095] In medical image recognition scenarios, such as determining the fracture grade corresponding to a fracture image, the image recognition method provided by embodiments of the present application can be employed, where the fracture grade is used to indicate the severity of the fracture. In some embodiments, the fracture image is an X-ray image. In some embodiments, fractures are classified into four grades, namely, grade one, grade two, grade three, and grade four, according to severity, with grade one being the least severe and grade four being the most severe. Grade one fractures are less severe and will heal on their own after a period of rest, so patients may not experience any discomfort. Therefore, patients with grade one fractures may not seek medical treatment, resulting in fewer fracture images corresponding to grade one fractures. Grade two fractures, while also less severe, are painful and uncomfortable, so patients will seek medical treatment, resulting in a higher number of fracture images corresponding to grade two fractures. Grade three and grade four fractures, due to their lower probability of occurrence, also have fewer fracture images corresponding to grade three and grade four fractures, respectively. Through the image recognition method provided in the embodiment of the present application, the computer equipment can train an image recognition model with a high recognition accuracy under the premise that the number of fracture images corresponding to first-degree fractures, second-degree fractures, third-degree fractures and fourth-degree fractures is unbalanced, and use the image recognition model to identify the fracture image and determine the fracture grade corresponding to the fracture image, thereby playing an auxiliary role in helping medical staff formulate fracture treatment plans.

[0096] In scenarios involving the recognition of material images, such as when identifying the cracking of metal materials, the image recognition method provided by the embodiments of the present application can be used. In some embodiments, the degree of cracking of metal materials is divided into primary cracking, secondary cracking, tertiary cracking, and quaternary cracking, wherein primary cracking has the lowest degree of cracking and quaternary cracking has the highest degree of cracking. For primary cracked metal materials, since their degree of cracking is low and not easily perceived by the human eye, there may be fewer metal material images corresponding to primary cracking. For secondary cracked metal materials, since their degree of cracking is higher than that of primary cracked metal materials, some experimenters can detect the cracking of the metal materials with the naked eye and often record the cracking of the metal materials by taking images. Therefore, the number of material images corresponding to secondary cracking is greater than that corresponding to primary cracking. For tertiary and quaternary cracked metal materials, since their degree of cracking is higher, there are more material images corresponding to each of them. Through the image recognition method provided in the embodiments of the present application, the computer device can train an image recognition model with a high recognition accuracy under the premise that the number of crack images corresponding to primary cracking, secondary cracking, tertiary cracking and quaternary cracking is unbalanced, and use the image recognition model to recognize the metal image and determine the crack level corresponding to the crack image.

[0097] In the scenario of identifying building images, the image recognition method provided by the embodiment of the present application can be used when determining the building type in the building image. In some embodiments, buildings include four types, which are divided into type A buildings, type B buildings, type C buildings and type D buildings according to their commonness, among which type A buildings are the least common and type D buildings are the most common. For type A buildings, due to their low commonness, the number of building images corresponding to type A buildings is also small. For type D buildings, due to their high commonness, the number of building images corresponding to type D buildings is also large. As for type B buildings and type C buildings, since their commonness is lower than that of type D buildings, the number of building images corresponding to type B buildings and type C buildings respectively is also less than the building image corresponding to type D buildings. Through the image recognition method provided by the embodiment of the present application, the computer device can train an image recognition model with a high recognition accuracy under the premise that the number of building images corresponding to type A buildings, type B buildings, type C buildings and type D buildings is unbalanced, and recognize the building images through the image recognition model to determine the building type corresponding to the building images.

[0098] In the scenario of identifying vehicle images, the image recognition method provided by the embodiment of the present application can be used when determining the type of vehicle in the vehicle image. In some embodiments, vehicles include four types, which are divided into Type A vehicles, Type B vehicles, Type C vehicles and Type D vehicles according to their commonness, among which Type A vehicles are the least common and Type D vehicles are the most common. For Type A vehicles, due to their low commonness, the number of vehicle images corresponding to Type A vehicles is also small. For Type D vehicles, due to their high commonness, the number of vehicle images corresponding to Type D vehicles is also large. As for Type B vehicles and Type C vehicles, since they are less common than Type D vehicles, the number of vehicle images corresponding to Type B vehicles and Type C vehicles respectively is less than the vehicle image corresponding to Type D vehicles. Through the image recognition method provided by the embodiment of the present application, the computer device can train an image recognition model with a high recognition accuracy under the premise that the number of vehicle images corresponding to Type A vehicles, Type B vehicles, Type C vehicles and Type D vehicles is unbalanced, and use the image recognition model to recognize the vehicle image and determine the vehicle type corresponding to the vehicle image.

[0099] In the embodiment of the present application, the computer device can implement the image recognition method provided in the embodiment of the present application with the help of an image recognition model. Figure 2 , the structure of the image recognition model provided in the embodiment of the present application is explained.

[0100] See also Figure 2 The image recognition model 200 includes a first branch network 201, a second branch network 202, and a third branch network 203. In some embodiments, each branch network has the same network structure.

[0101] The first branch network 201 includes a first sampler 2011, a first feature extraction layer 2012, a second feature extraction layer 2013, a pooling layer 2014, and a fully connected layer 2015. The first sampler 2011 is used to sample the sample dataset and input the sampled first sample image into the first feature extraction layer 2012. The first feature extraction layer 2012 and the second feature extraction layer 2013 perform feature extraction on the first sample image to obtain a first feature map of the first sample image. The first feature map is input into the pooling layer 2014, which performs pooling processing on the first feature map. The pooled first feature map is then input into the fully connected layer 2015 to obtain features of the first sample image. In some embodiments, the fully connected layer 2015 of the first branch network 201 is further connected to the classifier of the first branch network 201. The classifier is used to predict a predicted label for the first sample image based on the features of the first sample image.

[0102] The second branch network 202 includes a first sampler 2021, a first feature extraction layer 2022, a second feature extraction layer 2023, a pooling layer 2024, and a fully connected layer 2025. The first sampler 2021 is used to sample the sample data set and input the sampled second sample image into the first feature extraction layer 2022. The first feature extraction layer 2022 and the second feature extraction layer 2023 perform feature extraction on the second sample image to obtain a second feature map of the second sample image. The second feature map is input into the pooling layer 2024, and the second feature map is pooled by the pooling layer 2024. The second feature map after pooling is input into the fully connected layer 2025 to obtain features of the second sample image. In some embodiments, the fully connected layer 2025 of the second branch network 202 is further connected to a classifier of the second branch network 202. The classifier is used to predict a predicted label of the second sample image based on the features of the second sample image.

[0103] The third branch network 203 includes a first sampler 2031, a first feature extraction layer 2032, a second feature extraction layer 2033, a pooling layer 2034, and a fully connected layer 2035. The first sampler 2031 is used to sample the sample data set and input the sampled third sample image into the first feature extraction layer 2032. The first feature extraction layer 2032 and the second feature extraction layer 2033 perform feature extraction on the third sample image to obtain a third feature map of the third sample image. The third feature map is input into the pooling layer 2034, and the pooling layer 2034 performs pooling on the third feature map. The pooled third feature map is then input into the fully connected layer 2035 to obtain features of the third sample image. In some embodiments, the fully connected layer 2035 of the third branch network 203 is further connected to a classifier of the third branch network 203. The classifier is used to predict a predicted label for the third sample image based on the features of the third sample image.

[0104] It should be noted that Figure 2 The structure of the image recognition model shown is only an example. In other possible implementations, the image recognition model may also have other structures, which is not limited in the embodiments of the present application.

[0105] After introducing the implementation environment, application scenarios and structure of the image recognition model of the embodiment of the present application, the image recognition method provided by the embodiment of the present application is described below. In the embodiment of the present application, the image recognition method provided by the embodiment of the present application can be implemented by a server or a terminal as the execution subject, or the image recognition method provided by the present application can be implemented through interaction between a terminal and a server, wherein the terminal is the terminal 110 in the above-mentioned implementation environment, and the server is the server 140 in the above-mentioned implementation environment. As for the interaction between the terminal and the server, that is, the terminal sends a sample data set to the server, the server trains the image recognition model, and the server returns the trained image recognition model to the terminal. The embodiment of the present application does not limit the execution subject. The following description will take the execution subject as the server as an example, see Figure 3 , methods include:

[0106] 301. The server uniformly samples a sample data set to obtain a first sample image, where the sample data set includes sample images of at least two categories.

[0107] Uniform sampling means that the probability of each sample image in the sample data set being sampled is the same. In some embodiments, the number of sample images of at least two categories in the sample data set is different. If the image recognition method is applied to the scenario of identifying medical images, taking the medical image as an example of a fracture image, the sample data set includes at least two categories of sample images, that is, fracture images corresponding to at least two fracture levels. For example, the sample data set includes fracture images corresponding to four fracture levels, and the number of fracture images corresponding to each fracture level is different.

[0108] 302. The server samples sample images of a first category in the sample data set based on first sampling probabilities of at least two categories to obtain second sample images, where the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set.

[0109] The first sampling probability is related to the category of the sample image. If the sample image set contains more sample images corresponding to a category, the first sampling probability corresponding to that category will be lower, which means that the probability of the sample images corresponding to that category being sampled is lower. Correspondingly, if the sample image set contains fewer sample images corresponding to another category, the first sampling probability corresponding to that category will be higher, which means that the probability of the sample images corresponding to that category being sampled is higher.

[0110] 303. The server samples sample images of the second category in the sample data set based on the second sampling probability to obtain third sample images, where at least two categories correspond to the same second sampling probability.

[0111] The second sampling probability corresponding to each category in the sample data set is the same, which means that in the process of collecting the third sample image, the probability of sampling from each category is the same.

[0112] 304. The server trains an image recognition model based on the first sample image, the second sample image, the third sample image, and reference labels of the sample images, where the reference labels are used to indicate categories of the corresponding sample images.

[0113] In some embodiments, the reference label is a label added by a technician to the sample image. If the sample image is a fracture image, the reference label is the fracture grade corresponding to the fracture image.

[0114] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. Using the sample images obtained by the three sampling methods to train the image recognition model can improve the image recognition accuracy of the image model.

[0115] It should be noted that the above steps 301-304 are a brief introduction to the image recognition method provided by the embodiment of the present application. The following will combine some examples to explain the image recognition method provided by the embodiment of the present application more clearly. It should be noted that the training process of the image recognition model is divided into an iterative process. In the following description, the iterative process of training the image recognition model is used as an example, or the execution subject is the server as an example, see Figure 4 , methods include:

[0116] 401. The server uniformly samples a sample data set to obtain a first sample image, where the sample data set includes sample images of at least two categories.

[0117] Depending on the application of the image recognition model, the server can obtain different sample data sets. For example, if the image recognition model is used to identify the fracture grade corresponding to a fracture image, the sample data set will include multiple fracture images corresponding to different fracture grades; if the image recognition model is used to identify cracks in metal materials, the sample data set will also include multiple metal images corresponding to different cracking degrees.

[0118] In one possible implementation, the server determines the probability of each sample image being sampled based on the number of sample images in the sample dataset, where each sample image has the same probability of being sampled. Based on the probability of each sample image being sampled, the server samples the sample dataset to obtain a first sample image. In some embodiments, the server can perform the above implementation using a first sampler, also referred to as a sample-wise uniform sampler.

[0119] For example, the server determines the probability of each sample image being sampled based on the number of sample images in the sample data set. Based on the probability of each sample image being sampled, the server sets a first target number of image numbers for each sample image, and the first target number is positively correlated with the probability of sampling corresponding to the sample image. Since the first sample image is a sample image obtained by the server through uniform sampling, that is, the probability of each sample image being sampled as the first sample image is the same, the server sets the same number of image numbers for each sample image. The server randomly determines a target image number from the multiple image numbers corresponding to the sample image, and obtains the sample image corresponding to the target image number as a first sample image.

[0120] For example, if the sample dataset includes 100 sample images, the server determines that the probability of each sample image being sampled is 1%, based on the number of sample images (100). Based on the 1% probability of each sample image being sampled, the server assigns one or more image numbers to each sample image. For example, the server assigns image numbers to each sample image based on the order in which the 100 sample images are arranged in the sample dataset, i.e., 1-100. The server randomly determines an image number from 1-100, such as 40, and obtains the sample image corresponding to image number 40 as a first sample image.

[0121] 402. The server samples sample images of a first category in the sample data set based on first sampling probabilities of at least two categories to obtain second sample images, where the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set.

[0122] The negative correlation between the first sampling probability and the number of sample images of the corresponding category in the sample data set means that if the number of sample images corresponding to a category in the sample image set is large, then the first sampling probability corresponding to that category will be lower, which means that the probability of sample images corresponding to that category being sampled will be lower. Correspondingly, if the number of sample images corresponding to another category in the sample image set is small, then the first sampling probability corresponding to that category will be higher, which means that the probability of sample images corresponding to that category being sampled will be higher.

[0123] In one possible implementation, the server determines first sampling probabilities corresponding to the at least two categories based on the number of sample images in the at least two categories in the sample dataset. The server determines a first category from the at least two categories based on the first sampling probabilities corresponding to each category. The server then randomly samples from the sample images belonging to the first category to obtain a second sample image.

[0124] Through this implementation, the second sample images collected by the server are more inclined to categories with a smaller number of sample images. Using the second sample images to train the image recognition model can improve the image recognition model's ability to recognize categories with a smaller number of sample images.

[0125] For example, the server determines the first sampling probability of at least two categories based on the number of sample images in at least two categories in the sample data set. Based on the probability of at least two categories being sampled, the server sets a second target number of first category numbers for each category. The second target number is positively correlated with the first sampling probability corresponding to each category, that is, if the first sampling probability of a category is higher, then the server sets more first category numbers for the category. If the first sampling probability of a category is smaller, then the server sets a smaller first category number for the category. The server randomly determines a first target category number from multiple first category numbers, and determines the category corresponding to the first target category number as the first category. The server randomly samples multiple sample images in the first category to obtain a second sample image.

[0126] For example, the sample data set includes four categories, wherein the first category includes 20 sample images, the second category includes 60 sample images, the third category includes 10 sample images, and the fourth category includes 10 sample images. The server determines the first sampling probability for each of the four categories based on the number of sample images in the four categories. In some embodiments, the server obtains the reciprocal of the sample images in each category, namely 1 / 20, 1 / 60, 1 / 10, and 1 / 10. The server normalizes the four reciprocals 1 / 20, 1 / 60, 1 / 10, and 1 / 10 to obtain the first sampling probability corresponding to each category, namely, the first sampling probability corresponding to the first category is 18.75%, the first sampling probability corresponding to the second category is 6.25%, the first sampling probability corresponding to the third category is 37.5%, and the first sampling probability corresponding to the fourth category is 37.5%. The server sets the second target number of first category numbers for each of the four categories based on the first sampling probabilities corresponding to the four categories. Since the first sampling probability corresponding to the first category is 18.75, the server can set 1875 first category numbers for the first category, for example, 0-1874 is the first category number of the first category. The first sampling probability corresponding to the second category is 6.25%, so the server can set 625 first category numbers for the second category, for example, 1875-2499. The first sampling probability corresponding to the third category is 37.5%, so the server can set 3750 first category numbers for the third category, for example, 2500-6249. The first sampling probability corresponding to the fourth category is 37.5%, so the server can set 3750 first category numbers for the fourth category, for example, 6250-9999. The server randomly determines a first target category first category number from multiple first category numbers 0-9999, for example, 4235. The server determines the category corresponding to the first target category first category number 4235 as the first category, that is, the third category is determined as the first category. The server randomly samples 10 sample images of the third category to obtain a second sample image.

[0127] 403. The server samples sample images of the second category in the sample data set based on the second sampling probability to obtain third sample images, where at least two categories correspond to the same second sampling probability.

[0128] In one possible implementation, the server determines a second category from at least two categories based on the second sampling probability. The server samples sample images from the second category to obtain a third sample image. In some embodiments, the server can perform the above embodiment using a third sampler, also referred to as a class-wise sampler.

[0129] In this embodiment, the sample images corresponding to each category have the same probability of being sampled, which can eliminate the impact of the imbalance in the number of sample images in at least two categories on the training of the image recognition model.

[0130] For example, the server determines a second sampling probability based on the number of categories in the sample data set. Based on the second sampling probability, the server assigns a second category number to each category. The server determines a second target category number from the plurality of second category numbers and determines the category corresponding to the second target category number as the second category. The server randomly samples the plurality of sample images in the second category to obtain a third sample image.

[0131] For example, the sample data set includes four categories. Based on the number of categories 4, the server determines that the second sampling probability is 25%, and the probability of the sample image in each category being sampled as the third sample image is the same. Based on the second sampling probability of 25%, the server sets 250 second category numbers for each category, for example, the second category numbers set for the first category are 0-249, the second category numbers set for the second category are 250-499, the second category numbers set for the third category are 500-749, and the second category numbers set for the fourth category are 750-999. The server randomly determines a second target category number from multiple second category numbers 0-999, for example, 463. The server determines the category corresponding to the second target category number 463 as the second category, that is, the second category is determined as the second category. The server randomly samples from multiple sample images of the second category to obtain a third sample image.

[0132] It should be noted that, in the above description, the server executes steps 401-403 in sequence as an example. In other possible implementations, the server can also execute the above steps 401-403 in any order, that is, the server can obtain the first sample image, the second sample image and the third sample image in any order. The embodiment of the present application does not limit the sampling order of each sample image.

[0133] Additionally, optionally, before executing steps 401-403 above, the server can also resize multiple sample images in the sample dataset to the target size. For example, the server can scale the multiple sample images in the sample dataset, enlarging sample images smaller than the target size and reducing sample images larger than the target size, thereby achieving resizing of the multiple sample images. In this way, the server can resize multiple sample images to the target size, facilitating training of the image recognition model.

[0134] 404. The server predicts labels for each sample image through each branch network of the image recognition model to obtain predicted labels for each sample image. Each branch network has the same network structure.

[0135] The predicted label indicates the predicted type of the corresponding sample image. If the sample image is a fracture image, the predicted label indicates the fracture grade of the sample image. If the sample image is a metal material image, the predicted label indicates the crack grade of the metal material image.

[0136] In one possible implementation, the server performs feature extraction on the first sample image through the first branch network to obtain the features of the first sample image. The server obtains the predicted label of the first sample image based on the features of the first sample image. The server performs feature extraction on the second sample image through the second branch network to obtain the features of the second sample image. The server obtains the predicted label of the second sample image based on the features of the second sample image. The server performs feature extraction on the third sample image through the third branch network to obtain the features of the third sample image. The server obtains the predicted label of the third sample image based on the features of the third sample image. In some embodiments, the first branch network is also called the conventional branch, which is responsible for network representation learning. The second branch network is also called the re-balancing branch, which is responsible for classifier learning. The third branch network is also called the regularization branch, which is responsible for learning category-related prior knowledge.

[0137] For example, the server performs convolution processing on the first sample image using the first feature extraction layer of the first branch network to obtain a first feature map of the first sample image. The server performs convolution processing on the first feature map using the second feature extraction layer of the first branch network to obtain a second feature map of the first sample image. The server performs pooling and fully connected processing on the second feature map of the first sample image using the pooling layer and fully connected layer of the first branch network to obtain features of the first sample image. The server uses the classifier of the first branch network to classify the first sample image based on the features of the first sample image to obtain a predicted label for the first sample image. In other words, the server uses the classifier of the first branch network to normalize the features of the first sample image to obtain a probability distribution column for the first sample image. The values ​​in the probability distribution column represent the probability that the first sample image corresponds to a label. In response to the probability corresponding to any value in the probability distribution column of the first sample image meeting a target probability condition, the server determines the label corresponding to the value as the predicted label for the first sample image. Wherein, the probability meeting the target probability condition means that the probability is greater than or equal to the target probability threshold, or that the probability corresponding to the any value is the largest among the probabilities corresponding to all values ​​in the probability distribution column of the first sample image.

[0138] The server performs convolution processing on the second sample image through the first feature extraction layer of the second branch network to obtain a second feature map of the second sample image. The server performs convolution processing on the second feature map through the second feature extraction layer of the second branch network to obtain a second feature map of the second sample image. The server performs pooling processing and full connection processing on the second feature map of the second sample image through the pooling layer and the fully connected layer of the second branch network to obtain the features of the second sample image. The server performs classification based on the features of the second sample image through the classifier of the second branch network to obtain a predicted label of the second sample image. That is, the server performs normalization processing on the features of the second sample image through the classifier of the second branch network to obtain a probability distribution column of the second sample image, and the numerical value in the probability distribution column represents the probability that the second sample image corresponds to a label. In response to the probability corresponding to any numerical value in the probability distribution column of the second sample image meeting the target probability condition, the server determines the label corresponding to the numerical value as the predicted label of the second sample image.

[0139] The server performs convolution processing on the third sample image through the first feature extraction layer of the third branch network to obtain a third feature map of the third sample image. The server performs convolution processing on the third feature map through the second feature extraction layer of the third branch network to obtain a second feature map of the third sample image. The server performs pooling processing and full connection processing on the second feature map of the third sample image through the pooling layer and the fully connected layer of the third branch network to obtain the features of the third sample image. The server performs classification based on the features of the third sample image through the classifier of the third branch network to obtain a predicted label of the third sample image. That is, the server performs normalization processing on the features of the third sample image through the classifier of the third branch network to obtain a probability distribution column of the third sample image, and the numerical value in the probability distribution column represents the probability that the third sample image corresponds to a label. In response to the probability corresponding to any numerical value in the probability distribution column of the third sample image meeting the target probability condition, the server determines the label corresponding to the numerical value as the predicted label of the third sample image.

[0140] In some embodiments, the first feature extraction layer of the first branch network is the backbone network of the first branch network, the first feature extraction layer of the second branch network is the backbone network of the second branch network, and the first feature extraction layer of the third branch network is the backbone network of the third branch network. The first feature extraction layer of the first branch network, the first feature extraction layer of the second branch network, and the first feature extraction layer of the third branch network share parameters. In other words, the backbone networks of the first branch network, the second branch network, and the third branch network share network parameters, which can reduce the number of parameters of the image recognition model and reduce the storage space occupied by the image recognition model. Optionally, the structure of the backbone network is ResNet-50 (Residual Network-50), VGG-16 (Visual Geometry Group-16, Super Resolution Test Sequence-16), or DenseNet-121 (Deep Network-121), etc., which is not limited in this embodiment of the present application.

[0141] 405. The server obtains predicted ranking information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image. The predicted ranking information is used to represent the correlation between different sample images.

[0142] In one possible implementation, the server concatenates the features of the first sample image, the features of the second sample image, and the features of the third sample image to obtain a concatenated feature, and performs full-connection processing and normalization processing on the concatenated feature to obtain predicted ranking information.

[0143] For example, the server obtains the predicted ranking information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image through the following process (1).

[0144] (1)

[0145] Among them, z is the predicted ranking information, Softmax() is the normalization function used for normalization processing, is the weight matrix of the fully connected processing, Cat() is the feature splicing function, which is used to splice the features. is the feature of the first sample image, is the feature of the second sample image, is the feature of the third sample image.

[0146] If the server uses vectors to represent features, such as To represent the characteristics of the first sample image ,use To represent the characteristics of the second sample image ,use To represent the characteristics of the third sample image The server uses the feature concatenation function Cat() to concatenate the features of the first sample image. , the features of the second sample image And the characteristics of the third sample image Perform splicing to obtain splicing features The server uses a fully connected weight matrix , for example , and the splicing feature Multiply to get the splicing features after full connection processing The server uses the normalization function Softmax() to process the spliced ​​features after full connection processing. Perform normalization to obtain predicted ranking information .

[0147] 406. The server obtains supervised ranking information based on the reference labels of each sample image.

[0148] In one possible implementation, the server divides the reference label of the first sample image by the fused label to obtain a first ranking label. The server divides the reference label of the second sample image by the fused label to obtain a second ranking label. The server divides the reference label of the third sample image by the fused label to obtain a third ranking label. The server combines the first ranking label, the second ranking label, and the third ranking label to obtain supervised ranking information. The fused label is the label obtained by adding the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

[0149] For example, the server obtains supervised ranking information based on the reference labels of each sample image using the following formula (2).

[0150] (2)

[0151] in, To supervise the sorting information, is the reference label of the first sample image, is the reference label of the second sample image, is the reference label of the third sample image, + + Fusion tag.

[0152] If the server uses numerical values ​​to represent reference labels, for example, when determining the fracture grade corresponding to a fracture image, the server can use the numerical value corresponding to the bone grade as the reference label. For example, a value of 1 can be used to represent a first-grade fracture, a value of 2 can be used to represent a second-grade fracture, a value of 3 can be used to represent a third-grade fracture, and a value of 4 can be used to represent a fourth-grade fracture. In some embodiments, the server can also use a value of 0 to represent no fracture.

[0153] If the first sample image is an image corresponding to a secondary fracture, then the reference label of the first sample image is 2, which is =2. If the second sample image is an image corresponding to a grade 3 fracture, then the reference label of the second sample image is also 3, which is =3. If the third sample image is an image without fracture, then the reference label of the first sample image is 0, which is = 0. The server determines the supervision sorting information through the above formula (2) =(0.4 0.6 0).

[0154] 407. The server trains the image recognition model based on the first difference information between the predicted label of each sample image and the corresponding reference label.

[0155] In one possible implementation, the server fuses first difference information between the predicted label of each sample image and the corresponding reference label to obtain a first loss value, and trains the image recognition model based on the first loss value.

[0156] For example, the server uses the first weight, the second weight, and the third weight to weightedly sum the first difference information between the predicted label of each sample image and the corresponding reference label to obtain a first loss value. The first weight is negatively correlated with the ratio of the number of iteration cycles of the image recognition model to the total number of iteration cycles. The sum of the second weight and the first weight is the target value, and the third weight is greater than the first weight.

[0157] For example, the server fuses the first difference information between the predicted label of each sample image and the corresponding reference label using the following formula (3) to obtain a first loss value.

[0158] (3)

[0159] in, is the first loss value, is the first weight, It is a cross entropy calculation function. After E() calculation, the first difference information between the predicted label and the corresponding reference label can be obtained. is the weight matrix of the classifier of the first branch network, is the feature of the first sample image, is the reference label of the first sample image, 1- is the second weight, is the weight matrix of the classifier of the second branch network, is the feature of the second sample image, is the reference label of the second sample image, is the third weight, is the weight matrix of the classifier of the third branch network, is the feature of the third sample image, is the reference label of the third sample image. In some implementations, the first weight The server determines the weight according to the following formula (4). In some embodiments, the third weight is a multiple of the first weight, that is, = ,in, >1.

[0160] (4)

[0161] in, is the number of iterations of the image recognition model, is the total number of iteration cycles.

[0162] Based on the first loss value, the server adjusts the model parameters of the image recognition model, specifically the parameters of the three branch networks. If the first feature extraction layers of the three branch networks share parameters, the server only needs to adjust the parameters of the second feature extraction layer, the fully connected layer, and the classifier.

[0163] In one possible implementation, the server trains the image recognition model based on the first difference information and the second difference information, where the second difference information is the difference information between the predicted ranking information and the supervised ranking information.

[0164] For example, the server obtains a first loss value based on the first difference information. The server obtains a second loss value based on the second difference information. The server performs a weighted summation of the first loss value and the second loss value to obtain a joint loss value. The server trains the image recognition model based on the joint loss value.

[0165] The method for obtaining the first loss value based on the first difference information by the server is the same as the inventive concept of the previous embodiment, and the implementation process is described in the relevant description of the previous embodiment. The method for obtaining the second loss value based on the second difference information by the server is described below.

[0166] In some embodiments, the server obtains the second loss value based on the second difference information using the following formula (5).

[0167] (5)

[0168] in, is the second loss value, () is the KL divergence, which can also express the degree of difference between the supervised sorting information and the predicted sorting information. The divergence is also the second difference information. is the supervised ranking information, and z is the predicted ranking information.

[0169] In some embodiments, the server performs weighted summation of the first loss value and the second loss value using the following formula (6) to obtain a joint loss value.

[0170] (6)

[0171] in, is the joint loss value, is the first loss weight, is the second loss weight, is the first loss value, The first loss weight and the second loss weight are set by the technicians according to the actual situation, such as setting them both to 1, or setting Set to 0.8, It is set to 0.7, etc., and this embodiment of the present application does not limit this.

[0172] Based on the joint loss, the server adjusts the model parameters of the image recognition model, specifically the parameters of the three branch networks. If the first feature extraction layer of the three branch networks shares parameters, the server only needs to adjust the parameters of the second feature extraction layer, the fully connected layer, and the classifier.

[0173] It should be noted that, in the process of adjusting the model parameters of the image recognition model based on the loss value, the server can use the gradient descent method, such as the batch gradient descent method (Batch Gradient Descent), the stochastic gradient descent method (Stochastic Gradient Descent), Adam (Adam) or the mini-batch gradient descent method (Mini-batch Gradient Descent), and the embodiments of the present application are not limited to this.

[0174] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0175] In the experiment, 2725 X-ray images were used as sample data sets for training the image recognition model. Among the 2725 X-ray images, 1785 were images without fractures, that is, normal images. 190 were fracture images corresponding to first-grade fractures, 452 were fracture images corresponding to second-grade fractures, 196 were fracture images corresponding to third-grade fractures, and 102 were fracture images corresponding to fourth-grade fractures. Figure 5 From left to right, the image shows an unfractured image, a fractured image corresponding to a first-degree fracture, a fractured image corresponding to a second-degree fracture, a fractured image corresponding to a third-degree fracture, and a fractured image corresponding to a fourth-degree fracture. In some embodiments, different fracture images may correspond to different fracture sites, such as the scaphoid, tibia, and fibula. These 2,725 images were annotated by experienced physicians, indicating the fracture type for each image. This process essentially adds reference labels to these 2,725 images.

[0176] Table 1 below compares the effects of using different branch network combinations for sampling and training image recognition models.

[0177] Table 1

[0178]

[0179] The effects of the image recognition model provided in the embodiments of the present application are further described below through Tables 2 to 5.

[0180] Table 2

[0181]

[0182] Table 3

[0183]

[0184] Table 4

[0185]

[0186] Table 5

[0187]

[0188] Among them, Table 2 is the recognition result of the vanilla model for fracture images. The vanilla model is a single-branch network. Table 3 is the recognition result of the label-distribution-aware margin (LDAM) model for fracture images. The label distribution awareness model is also a unit branch network. Table 4 is the recognition result of the bilateral-branch network (TBN) for fracture images. Table 5 is the recognition result of the three-branch network provided in the embodiment of the present application for fracture images. When viewing Tables 2 to 5 above, the labels in the first column of the table are reference labels, and the labels in the last row are predicted labels, that is, the marked labels. The number corresponding to a reference label and a predicted label represents the number of images that should correspond to the reference label and are identified as the predicted label. For example, for the reference label "normal" in the first column of Table 5, the number corresponding to the predicted label "normal" is 18, which means that 18 sample images with the reference label "normal" are identified as the predicted label "normal"; the number corresponding to the predicted label "first-grade fracture" is 0, which means that no sample image with the reference label "normal" is identified as the predicted label "first-grade fracture".

[0189] As can be seen from Tables 1-5, when only the first loss value is used to train the image recognition model, using any one or two of the first, second, and third branch networks individually is not as effective as using all three together. The effect of using both the first and second loss values ​​to train the image recognition model is better than using only the first loss value to train the image recognition model.

[0190] In addition, for the above formula (3) and In terms of the relationship between It is realized by gradually increasing from 0-0.5. = 0.5, the image recognition model has the highest recognition accuracy. As the value of continues to increase, the recognition accuracy of the image recognition model will decrease.

[0191] It should be noted that the image recognition model provided in this embodiment includes three branch networks, hence the name Triplet-branch Network (TBN). This image recognition model runs on PyTorch, and the backbone network of the three branch networks shares network parameters, reducing the complexity of the image recognition model. The image recognition model is trained using mini-batches, with each batch consisting of 16 sample images. The initial learning rate is set to 0.001, and the Adam algorithm is used to update the model parameters. The entire image recognition model training process consists of 100 cycles.

[0192] In addition, Table 6 provides a comparison of the effects of the image recognition model provided in the embodiment of the present application and other advanced algorithms on the APTOS 2019 dataset.

[0193] Table 6

[0194]

[0195] Among them, advanced algorithm 1 is DLI (Diabetic Retinopathy Detection Through Integration of Deep Learning Classication Framework), advanced algorithm 2 is CANet (Cross-disease Attention Network), and advanced algorithm 3 is GREEN (Graph Residual Re-ranking Network).

[0196] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. Using the sample images obtained by the three sampling methods to train the image recognition model can improve the image recognition accuracy of the image model.

[0197] In addition to the above steps 401-407, the present embodiment also provides an image recognition method, see Figure 6 , methods include:

[0198] 601. The server obtains an image to be recognized.

[0199] The image to be identified is any one of a fracture image, a metal material image, a building image or a vehicle image, which is not limited in the embodiment of the present application.

[0200] In a possible implementation, the terminal sends an image to be recognized to a server, and the server acquires the image.

[0201] The above implementation is described below through two examples.

[0202] Example 1: A terminal runs an image recognition application, allowing the user to select an image to be recognized. In response to selecting an image, the terminal sends an image recognition request to a server, including the image to be recognized. The server receives the image recognition request and retrieves the image to be recognized from it.

[0203] For example, see Figure 7The terminal displays an image recognition interface 701 of an image application, which includes an image selection control 702. In response to a click on image selection control 702, the terminal displays an image selection box 703 on the image recognition interface, which displays multiple images. In response to a selection of any image in the image selection box, the terminal sends an image recognition request to the server. The request carries the selected image, which is the image to be recognized. The server receives the request and retrieves the image from the request.

[0204] Example 2: A camera is installed on the terminal or is connected to the camera. In response to the camera capturing an image, the terminal sends an image recognition request to the server. The image recognition request includes the image to be recognized. The server receives the image recognition request and retrieves the image from the request.

[0205] For example, suppose the imaging device is an X-ray imaging device used to capture images of a patient's fractures. In response to the imaging device capturing any fracture image, the terminal sends an image recognition request to the server. This request includes the fracture image, which is the image to be recognized. The server receives the image recognition request and retrieves the fracture image from the request. The server then performs image recognition on the fracture image to determine the fracture grade corresponding to the fracture image.

[0206] 602. The server calls the image recognition model to recognize the image and obtain a predicted label for the image.

[0207] The image recognition model is trained based on a first sample image, a second sample image, a third sample image, and reference labels for each sample image, where the reference labels are used to indicate the category of the corresponding sample image. The first sample image is obtained by uniformly sampling multiple sample images in a sample data set, where the sample data set includes sample images of at least two categories. The second sample image is obtained by sampling sample images of a first category in the sample data set based on a first sampling probability for at least two categories. The third sample image is obtained by sampling sample images of a second category in the sample data set based on a second sampling probability. The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and at least two categories correspond to the same second sampling probability.

[0208] The training process of the image recognition model is described in the above steps 401-407 and will not be repeated here.

[0209] In one possible implementation, the image recognition model includes multiple branch networks, each with the same network structure. The server uses each branch network to predict labels for the image, generating a probability distribution column for the image. The values ​​in the probability distribution column represent the probability that the image corresponds to a label. In response to any value in the probability distribution column indicating a probability that meets a target probability condition, the server determines the label corresponding to that value as the predicted label.

[0210] For example, the server performs feature extraction on the image through the first branch network to obtain the first image feature of the image. Based on the first image feature, the server obtains the first probability distribution column of the image. The server performs feature extraction on the image through the second branch network to obtain the second image feature of the image. Based on the second image feature, the server obtains the second probability distribution column of the image. The server performs feature extraction on the image through the third branch network to obtain the third image feature of the image. Based on the third image feature, the server obtains the third probability distribution column of the image. The server adds the first probability distribution column, the second probability distribution column, and the third probability distribution column to obtain the probability distribution column of the image. In response to the probability represented by any numerical value in the probability distribution column meeting the target probability condition, the server determines the label corresponding to the numerical value as the predicted label.

[0211] In a possible implementation, after step 603 , the server can send the predicted label of the image to the terminal, and the user can view the predicted label of the image through the terminal.

[0212] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. The sample images obtained by the three sampling methods are used to recognize images, and have a higher image recognition accuracy.

[0213] Figure 8 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application, see Figure 8 The device includes: a first sampling module 801, a second sampling module 802, a third sampling module 803 and a training module 804.

[0214] The first sampling module 801 is configured to uniformly sample a sample data set to obtain a first sample image. The sample data set includes sample images of at least two categories.

[0215] The second sampling module 802 is used to sample sample images of a first category in the sample data set based on first sampling probabilities of at least two categories to obtain second sample images, where the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set.

[0216] The third sampling module 803 is configured to sample sample images of the second category in the sample data set based on the second sampling probability to obtain third sample images, wherein at least two categories correspond to the same second sampling probability.

[0217] The training module 804 is used to train the image recognition model based on the first sample image, the second sample image, the third sample image and the reference labels of the sample images, where the reference labels are used to indicate the categories of the corresponding sample images.

[0218] In one possible implementation, the image recognition model includes multiple branch networks, each having the same network structure. The training module 804 is configured to predict labels for each sample image using each branch network to obtain predicted labels for each sample image. The image recognition model is trained based on first difference information between the predicted labels for each sample image and the corresponding reference labels.

[0219] In one possible implementation, the training module 804 is configured to fuse first difference information between the predicted label of each sample image and the corresponding reference label to obtain a first loss value.

[0220] Based on the first loss value, the image recognition model is trained.

[0221] In one possible implementation, the training module 804 is used to use a first weight, a second weight, and a third weight to weightedly sum the first difference information between the predicted label of each sample image and the corresponding reference label to obtain a first loss value. The first weight is negatively correlated with the ratio of the number of iterative cycles of the image recognition model to the total number of iterative cycles. The sum of the second weight and the first weight is the target value, and the third weight is greater than the first weight.

[0222] In one possible implementation, the training module 804 is configured to extract features from the first sample image using the first branch network to obtain features of the first sample image, and obtain a predicted label for the first sample image based on the features of the first sample image.

[0223] The second branch network is used to extract features of the second sample image to obtain features of the second sample image. Based on the features of the second sample image, a predicted label of the second sample image is obtained.

[0224] The third branch network is used to extract features of the third sample image to obtain features of the third sample image, and a predicted label of the third sample image is obtained based on the features of the third sample image.

[0225] In one possible implementation, the apparatus further includes:

[0226] The predicted sorting information acquisition module is used to acquire predicted sorting information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, where the predicted sorting information is used to represent the correlation between different sample images.

[0227] The supervised sorting information acquisition module is used to obtain supervised sorting information based on the reference labels of each sample image.

[0228] The training module 804 is used to train the image recognition model based on the first difference information and the second difference information, where the second difference information is the difference information between the predicted ranking information and the supervised ranking information.

[0229] In a possible implementation, the prediction ranking information acquisition module is configured to splice features of the first sample image, features of the second sample image, and features of the third sample image to obtain spliced ​​features.

[0230] The splicing features are fully connected and normalized to obtain the predicted ranking information.

[0231] In a possible implementation, the supervised ranking information acquisition module is configured to divide the reference label of the first sample image by the fusion label to obtain a first ranking label.

[0232] The reference label of the second sample image is divided by the fusion label to obtain the second ranking label.

[0233] The reference label of the third sample image is divided by the fusion label to obtain the third ranking label.

[0234] The first ranking label, the second ranking label, and the third ranking label are combined to obtain supervised ranking information.

[0235] The fusion label is a label obtained by adding the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

[0236] In one possible implementation, the training module 804 is configured to obtain a first loss value based on the first difference information.

[0237] Based on the second difference information, a second loss value is obtained.

[0238] The first loss value and the second loss value are weightedly summed to obtain the joint loss value.

[0239] Based on the joint loss value, the image recognition model is trained.

[0240] It should be noted that the image recognition device provided in the above embodiment is only illustrated by the division of the above functional modules when performing image recognition. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device provided in the above embodiment and the image recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0241] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. Using the sample images obtained by the three sampling methods to train the image recognition model can improve the image recognition accuracy of the image model.

[0242] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application, see Figure 9 The device includes: an image acquisition module 901 and an image recognition module 902.

[0243] The image acquisition module 901 is used to acquire an image to be identified.

[0244] The image recognition module 902 is used to call the image recognition model to recognize the image and obtain the predicted label of the image.

[0245] The image recognition model is trained based on the first sample image, the second sample image, the third sample image, and reference labels of each sample image, and the reference labels are used to represent the category of the corresponding sample image.

[0246] The first sample image is obtained by uniformly sampling multiple sample images in a sample dataset, where the sample dataset includes sample images of at least two categories. The second sample image is obtained by sampling sample images of the first category in the sample dataset based on a first sampling probability of the at least two categories. The third sample image is obtained by sampling sample images of the second category in the sample dataset based on a second sampling probability.

[0247] The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and at least two categories correspond to the same second sampling probability.

[0248] In one possible implementation, the image recognition model includes multiple branch networks, each branch network has the same network structure. The image recognition module 902 is used to predict the label of the image through each branch network to obtain a probability distribution column of the image. The numerical value in the probability distribution column is used to represent the probability that the image corresponds to a label.

[0249] In response to the probability represented by any numerical value in the probability distribution column meeting the target probability condition, the label corresponding to the any numerical value is determined as the predicted label.

[0250] In one possible implementation, the image recognition module 902 is configured to extract features from the image using a convolutional layer of the first branch network to obtain a first image feature of the image, and obtain a first probability distribution column of the image based on the first image feature.

[0251] The second branch network is used to extract features from the image to obtain a second image feature of the image. Based on the second image feature, a second probability distribution column of the image is obtained.

[0252] The third branch network is used to extract features from the image to obtain a third image feature of the image. Based on the third image feature, a third probability distribution column of the image is obtained.

[0253] The first probability distribution column, the second probability distribution column, and the third probability distribution column are added together to obtain a probability distribution column of the image.

[0254] It should be noted that the image recognition device provided in the above embodiment is only illustrated by the division of the above functional modules when performing image recognition. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device provided in the above embodiment and the image recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0255] Through the technical solution provided by the embodiment of the present application, under the premise that there are at least two categories of sample images in the sample data set, different methods can be used to sample in the sample data set to obtain a first sample image, a second sample image, and a third sample image. Among them, the first sample image is a sample image obtained by uniform sampling, so the first sample image can be used to train the image recognition model's basic recognition ability of the image. The second sample image is a sample image obtained based on the first sampling probability of at least two categories, and the sampling method indicated by the first sampling probability is that the probability of the category with a smaller number of samples is higher. After the image recognition model is trained with the second sample image, the recognition result of the image recognition model is biased towards the category corresponding to the sample image with a smaller number of samples. The third sample image is a sample image obtained by sampling after setting the same sampling probability for each category. The image recognition model is trained with the third sample image, and the image recognition model obtained has a more balanced image recognition ability for different categories. The sample images obtained by the three sampling methods are used to recognize images, and have a higher image recognition accuracy.

[0256] The present application embodiment provides a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is first introduced below:

[0257] Figure 10 1 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 1000 may be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal 1000 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0258] Typically, the terminal 1000 includes: one or more processors 1001 and one or more memories 1002 .

[0259] Processor 1001 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1001 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0260] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one computer program, which is used to be executed by the processor 1001 to implement the image recognition method provided in the method embodiment of the present application.

[0261] In some embodiments, terminal 1000 may optionally include a peripheral device interface 1003 and at least one peripheral device. Processor 1001, memory 1002, and peripheral device interface 1003 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1003 via a bus, signal lines, or circuit boards. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, a positioning assembly 1008, and a power supply 1009.

[0262] The peripheral device interface 1003 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0263] RF circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1004 communicates with communication networks and other communication devices using electromagnetic signals. RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. RF circuit 1004 may optionally include an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on.

[0264] Display screen 1005 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If display screen 1005 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 1005. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0265] The camera assembly 1006 is used to collect images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal.

[0266] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication.

[0267] The positioning component 1008 is used to locate the current geographical location of the terminal 1000 to implement navigation or LBS (Location Based Service).

[0268] The power supply 1009 is used to supply power to various components in the terminal 1000. The power supply 1009 can be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0269] In some embodiments, the terminal 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .

[0270] The acceleration sensor 1011 can detect the magnitude of acceleration on three coordinate axes of a coordinate system established with the terminal 1000 .

[0271] The gyroscope sensor 1012 can detect the body direction and rotation angle of the terminal 1000 . The gyroscope sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal 1000 .

[0272] The pressure sensor 1013 can be provided on the side frame of the terminal 1000 and / or below the display screen 1005. When the pressure sensor 1013 is provided on the side frame of the terminal 1000, it can detect the user's gripping signal of the terminal 1000, and the processor 1001 can perform left and right hand recognition or shortcut operations based on the gripping signal collected by the pressure sensor 1013. When the pressure sensor 1013 is provided below the display screen 1005, the processor 1001 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1005.

[0273] The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity based on the collected fingerprint.

[0274] The optical sensor 1015 is used to collect ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015.

[0275] The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000 .

[0276] Those skilled in the art will understand that Figure 10 The structure shown in the figure does not constitute a limitation on the terminal 1000, and the terminal 1000 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0277] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0278] Figure 11 This is a structural diagram of a server provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein the one or more memories 1102 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1101 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 1100 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1100 may also include other components for implementing device functions, which will not be described in detail here.

[0279] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program. The computer program can be executed by a processor to perform the image recognition method described in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0280] In an exemplary embodiment, a computer program product or computer program is also provided, which includes a program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned image recognition method.

[0281] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0282] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0283] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image recognition method, characterized in that: The method comprises: Uniformly sampling a sample data set to obtain a first sample image, wherein the sample data set includes sample images of at least two categories; Based on the first sampling probabilities of the at least two categories, sampling sample images of the first category in the sample data set to obtain second sample images, wherein the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set; Based on the second sampling probability, sampling the sample images of the second category in the sample data set to obtain a third sample image, wherein the at least two categories correspond to the same second sampling probability; Performing feature extraction on the first sample image, the second sample image, and the third sample image through a first branch network, a second branch network, and a third branch network in the image recognition model, respectively, to obtain features of the first sample image, features of the second sample image, and features of the third sample image, wherein the first branch network, the second branch network, and the third branch network have the same network structure; Based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, respectively obtain a predicted label for the first sample image, a predicted label for the second sample image, and a predicted label for the third sample image; acquiring predicted sorting information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, where the predicted sorting information is used to represent the correlation between different sample images; Obtaining supervised sorting information based on a reference label of the first sample image, a reference label of the second sample image, and a reference label of the third sample image, wherein the reference label is used to indicate a category of a corresponding sample image; The image recognition model is trained based on second difference information and first difference information between the predicted label of each sample image and the corresponding reference label, where the second difference information is difference information between the predicted ranking information and the supervised ranking information.

2. The method according to claim 1, characterized in that The method further comprises: Fusing first difference information between the predicted labels of each sample image and the corresponding reference labels to obtain a first loss value; The image recognition model is trained based on the first loss value.

3. The method according to claim 2, characterized in that The fusing first difference information between the predicted labels of the respective sample images and the corresponding reference labels to obtain a first loss value includes: Using the first weight, the second weight and the third weight, the first difference information between the predicted label of each sample image and the corresponding reference label is weighted and summed to obtain the first loss value, the first weight is negatively correlated with the ratio of the number of iterative cycles of the image recognition model to the total number of iterative cycles, the sum of the second weight and the first weight is the target value, and the third weight is greater than the first weight.

4. The method according to claim 1, wherein The acquiring the predicted sorting information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image includes: splicing the features of the first sample image, the features of the second sample image, and the features of the third sample image to obtain a spliced ​​feature; Fully connected and normalized processing is performed on the splicing features to obtain the predicted ranking information.

5. The method according to claim 1, wherein The acquiring supervised sorting information based on the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image includes: Dividing the reference label of the first sample image by the fusion label to obtain a first ranking label; Dividing the reference label of the second sample image by the fusion label to obtain a second ranking label; Dividing the reference label of the third sample image by the fusion label to obtain a third ranking label; Combining the first sorting label, the second sorting label, and the third sorting label to obtain the supervised sorting information; The fusion label is a label obtained by adding the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

6. The method according to claim 1, characterized in that The training of the image recognition model based on the second difference information and the first difference information between the predicted label of each sample image and the corresponding reference label includes: Obtaining a first loss value based on the first difference information; Obtaining a second loss value based on the second difference information; Performing a weighted summation on the first loss value and the second loss value to obtain a joint loss value; The image recognition model is trained based on the joint loss value.

7. An image recognition method, characterized in that: The method further comprises: Obtaining an image to be recognized; Calling an image recognition model to recognize the image and obtain a predicted label for the image; The image recognition model is trained based on the second difference information and the first difference information between the predicted label of each sample image and the corresponding reference label; The sample images are divided into a first sample image, a second sample image, and a third sample image. The first sample image is obtained by uniformly sampling multiple sample images in a sample data set, and the sample data set includes sample images of at least two categories. The second sample image is obtained by sampling sample images of the first category in the sample data set based on a first sampling probability of the at least two categories. The third sample image is obtained by sampling sample images of the second category in the sample data set based on a second sampling probability. The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and the at least two categories correspond to the same second sampling probability. The reference label is used to represent the category of the corresponding sample image; the predicted label of the first sample image, the predicted label of the second sample image, and the predicted label of the third sample image are obtained based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, respectively; The features of the first sample image, the features of the second sample image, and the features of the third sample image are obtained by extracting features of the first sample image, the second sample image, and the third sample image respectively through a first branch network, a second branch network, and a third branch network in an image recognition model, wherein the first branch network, the second branch network, and the third branch network have the same network structure; The second difference information is the difference information between the predicted sorting information and the supervised sorting information; the predicted sorting information is used to represent the correlation between different sample images, and the predicted sorting information is obtained based on the features of the first sample image, the features of the second sample image, and the features of the third sample image; the supervised sorting information is obtained based on the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

8. The method according to claim 7, characterized in that Calling the image recognition model to recognize the image and obtain a predicted label for the image includes: Performing label prediction on the image through each branch network in the image recognition model to obtain a probability distribution column of the image, wherein the numerical value in the probability distribution column is used to represent the probability that the image corresponds to a label; In response to a probability represented by any numerical value in the probability distribution column meeting a target probability condition, a label corresponding to the any numerical value is determined as the predicted label.

9. An image recognition device, characterized in that: The device comprises: a first sampling module, configured to uniformly sample a sample data set to obtain a first sample image, wherein the sample data set includes sample images of at least two categories; a second sampling module, configured to sample sample images of a first category in the sample data set based on first sampling probabilities of the at least two categories to obtain second sample images, wherein the first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set; a third sampling module, configured to sample sample images of the second category in the sample data set based on a second sampling probability to obtain a third sample image, wherein the at least two categories correspond to the same second sampling probability; a training module, configured to extract features from the first sample image, the second sample image, and the third sample image, respectively, through a first branch network, a second branch network, and a third branch network in the image recognition model, to obtain features of the first sample image, features of the second sample image, and features of the third sample image, wherein the first branch network, the second branch network, and the third branch network have the same network structure; The training module is further configured to respectively obtain a predicted label of the first sample image, a predicted label of the second sample image, and a predicted label of the third sample image based on features of the first sample image, features of the second sample image, and features of the third sample image; a predicted ranking information acquisition module, configured to acquire predicted ranking information based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, wherein the predicted ranking information is used to represent the correlation between different sample images; a supervised sorting information acquisition module, configured to acquire supervised sorting information based on a reference label of the first sample image, a reference label of the second sample image, and a reference label of the third sample image, wherein the reference label is used to indicate a category of the corresponding sample image; The training module is also used to train the image recognition model based on the second difference information and the first difference information between the predicted label of each sample image and the corresponding reference label, where the second difference information is the difference information between the predicted ranking information and the supervised ranking information.

10. The device according to claim 9, characterized in that The training module is used to: Fusing first difference information between the predicted labels of each sample image and the corresponding reference labels to obtain a first loss value; The image recognition model is trained based on the first loss value.

11. The device according to claim 10, characterized in that The training module is used to: Using the first weight, the second weight and the third weight, the first difference information between the predicted label of each sample image and the corresponding reference label is weighted and summed to obtain the first loss value, the first weight is negatively correlated with the ratio of the number of iterative cycles of the image recognition model to the total number of iterative cycles, the sum of the second weight and the first weight is the target value, and the third weight is greater than the first weight.

12. The device according to claim 9, characterized in that The prediction ranking information acquisition module is used to: splicing the features of the first sample image, the features of the second sample image, and the features of the third sample image to obtain a spliced ​​feature; Fully connected and normalized processing is performed on the splicing features to obtain the predicted ranking information.

13. The device according to claim 9, characterized in that The supervised sorting information acquisition module is used to: Dividing the reference label of the first sample image by the fusion label to obtain a first ranking label; Dividing the reference label of the second sample image by the fusion label to obtain a second ranking label; Dividing the reference label of the third sample image by the fusion label to obtain a third ranking label; Combining the first sorting label, the second sorting label, and the third sorting label to obtain the supervised sorting information; The fusion label is a label obtained by adding the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

14. The device according to claim 9, characterized in that The training module is used to: Obtaining a first loss value based on the first difference information; Obtaining a second loss value based on the second difference information; Performing a weighted summation on the first loss value and the second loss value to obtain a joint loss value; The image recognition model is trained based on the joint loss value.

15. An image recognition device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be identified; An image recognition module is used to call an image recognition model to recognize the image and obtain a predicted label for the image; The image recognition model is trained based on the second difference information and the first difference information between the predicted label of each sample image and the corresponding reference label; The sample images are divided into a first sample image, a second sample image, and a third sample image. The first sample image is obtained by uniformly sampling multiple sample images in a sample data set, and the sample data set includes sample images of at least two categories. The second sample image is obtained by sampling sample images of the first category in the sample data set based on a first sampling probability of the at least two categories. The third sample image is obtained by sampling sample images of the second category in the sample data set based on a second sampling probability. The first sampling probability is negatively correlated with the number of sample images of the corresponding category in the sample data set, and the at least two categories correspond to the same second sampling probability. The reference label is used to represent the category of the corresponding sample image; the predicted label of the first sample image, the predicted label of the second sample image, and the predicted label of the third sample image are obtained based on the features of the first sample image, the features of the second sample image, and the features of the third sample image, respectively; The features of the first sample image, the features of the second sample image, and the features of the third sample image are obtained by extracting features of the first sample image, the second sample image, and the third sample image respectively through a first branch network, a second branch network, and a third branch network in an image recognition model, wherein the first branch network, the second branch network, and the third branch network have the same network structure; The second difference information is the difference information between the predicted sorting information and the supervised sorting information; the predicted sorting information is used to represent the correlation between different sample images, and the predicted sorting information is obtained based on the features of the first sample image, the features of the second sample image, and the features of the third sample image; the supervised sorting information is obtained based on the reference label of the first sample image, the reference label of the second sample image, and the reference label of the third sample image.

16. The device according to claim 15, characterized in that The image recognition module is used to: Performing label prediction on the image through each branch network in the image recognition model to obtain a probability distribution column of the image, wherein the numerical value in the probability distribution column is used to represent the probability that the image corresponds to a label; In response to a probability represented by any numerical value in the probability distribution column meeting a target probability condition, a label corresponding to the any numerical value is determined as the predicted label.

17. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the image recognition method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the image recognition method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product includes program code, and a processor of a computer device loads and executes the program code, so that the computer device implements the image recognition method according to any one of claims 1 to 8.

Images