Model training method for object classification, object classification method and device

Through sampling and feature fusion of the two branch networks, combined with similarity difference training, the problem of time-consuming and geographical limitations of artificially identifying image lesion levels in the medical field is solved, and high-accuracy automated grading evaluation is achieved, reducing costs and burdens.

CN114372564BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210033666.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-12
Publication Date
2024-11-26
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

The prior art artificial identification of image lesion levels in the medical field has problems such as time-consuming, geographical limitations and long queues of patients, and artificial intelligence has not yet effectively solved these problems in the development of medical grading methods.

Method used

A model training method is adopted, using two branch networks to sample and feature extraction of the sample data sets respectively. By fusion of features and labels, positive and negative samples are determined, and similarity differences are used for network training to form a target model. Implement object hierarchy.

Benefits of technology

It improves the accuracy of subject grading, reduces the burden on clinicians to read the film, realizes automated lesion-level evaluation, reduces costs, and improves the convenience and accuracy of grading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114372564B_ABST
    Figure CN114372564B_ABST
Patent Text Reader

Abstract

The present application provides a model training method, an object classification method and a device for object classification. It relates to technical fields such as machine learning, big data, and cloud medical care. The sample data set is sampled by two branch networks according to different category sampling rates, and the first sample feature and the second sample feature of each sampled data are obtained respectively, and the two sample features are fused to obtain at least two fused features, and the two sample labels are fused to obtain at least two fused labels corresponding to the at least two fused features to improve the richness of the sample features; the positive sample features and the negative sample features in the at least two fused features are determined, and the first similarity between the anchor feature and the positive sample feature and the second similarity between the anchor feature and the negative sample feature are determined, so as to train the network to obtain the target model based on the difference between the two similarities; the network is compared and learned based on the positive and negative sample pairs, so as to improve the accuracy of object classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to technical fields such as artificial intelligence, machine learning, and cloud medicine, and the present application relates to a model training method, an object classification method, and a device for object classification. Background Art

[0002] With the development of artificial intelligence technology, image recognition functions using network models have been realized, such as identifying the content of an image. However, in the medical field, doctors are still required to manually identify the patient's image to determine the level of the patient's lesion.

[0003] However, manual identification has many limitations. For example, manual identification takes a long time, patients have to wait in line for a long time, and due to geographical restrictions, patients need to register in advance and go to the hospital for diagnosis. With the development of artificial intelligence in the medical field, the object classification method using network models has become a hot issue that technicians in this field need to study urgently. Summary of the invention

[0004] The present application provides a model training method, an object classification method and a device for object classification, and the technical scheme is as follows:

[0005] In one aspect, a model training method for object classification is provided, wherein an initial model to be trained includes a first branch network and a second branch network, and the method includes:

[0006] Sampling the sample data set based on the first branch network and extracting a first sample feature of the sampled first sample data, and sampling the sample data set based on the second branch network and extracting a second sample feature of the sampled second sample data;

[0007] The first sample feature and the second sample feature are respectively fused to obtain at least two fused features, and the first sample label and the second sample label are fused to obtain at least two fused labels corresponding to the at least two fused features;

[0008] Determine a positive sample feature and a negative sample feature in the at least two fused features based on the anchor point label corresponding to the anchor point feature and the at least two fused labels;

[0009] Determine a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and train the first branch network and the second branch network based on the difference between the first similarity and the second similarity to obtain a trained target model;

[0010] The sample data set includes at least two categories of samples, the category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of at least two categories of samples with different sample sizes in the sample data set; the first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label; the anchor feature is any one of the first sample feature and the second sample feature.

[0011] In another aspect, a method for classifying an object is provided, the method comprising:

[0012] In response to a grading request of any requesting device, based on a target image of any object, calling a target model to obtain a level of the object corresponding to the target image, wherein the grading request is used to request to return the level of the object corresponding to the target image;

[0013] Returning the level of the object corresponding to the target image to any requesting device;

[0014] Wherein, the target model is trained based on the above-mentioned model training method for object classification.

[0015] On the other hand, a model training device for object classification is provided, wherein an initial model to be trained includes a first branch network and a second branch network, and the device includes:

[0016] an extraction module, configured to sample the sample data set based on the first branch network and extract a first sample feature of the sampled first sample data, and to sample the sample data set based on the second branch network and extract a second sample feature of the sampled second sample data;

[0017] a fusion module, configured to fuse the first sample feature and the second sample feature respectively to obtain at least two fused features, and to fuse the first sample label and the second sample label to obtain at least two fused labels corresponding to the at least two fused features;

[0018] A determination module, configured to determine a positive sample feature and a negative sample feature in the at least two fused features based on an anchor point label corresponding to the anchor point feature and the at least two fused labels;

[0019] A training module, used to determine a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and based on the difference between the first similarity and the second similarity, train the first branch network and the second branch network to obtain a trained target model;

[0020] The sample data set includes at least two categories of samples, the category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of at least two categories of samples with different sample sizes in the sample data set; the first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label; the anchor feature is any one of the first sample feature and the second sample feature.

[0021] In one possible implementation, the determination module is used to determine the similarity between the anchor point label and each of the at least two fused features, respectively, to obtain at least two similarities; the fused feature corresponding to the third similarity among the at least two similarities is used as the positive sample feature, and the fused feature corresponding to the fourth similarity among the at least two similarities is used as the negative sample feature, and the third similarity is greater than the fourth similarity.

[0022] In one possible implementation, the training module is used to iteratively train the first branch network and the second branch network when the difference between the second similarity and the first similarity is not higher than a target value, and stop training until the second similarity is higher than the target value of the first similarity to obtain the target model.

[0023] In a possible implementation, the fusion module is used to obtain at least two fusion parameter pairs, each fusion parameter pair includes two random parameters, and the value range of the two random parameters is greater than 0 and less than 1; for each fusion parameter pair, based on the two random parameters included in the fusion parameter pair, the first sample feature and the second sample feature are fused respectively to obtain a fusion feature corresponding to the fusion parameter pair, and the first sample label and the second sample label are fused to obtain a fusion label of the fusion feature corresponding to the fusion parameter pair.

[0024] In one possible implementation, the extraction module is used to uniformly sample the sample data set based on the first branch network to obtain the first sample data, and extract a first sample feature of the first sample data; determine the sample frequency of each category in the sample data set based on the second branch network, reversely sample the samples of the at least two categories based on the sample frequency of each category to obtain the second sample data, and extract a second sample feature of the second sample data, wherein the reverse sampling means that the sample frequency of at least two categories is inversely proportional to the sampling rate, and the sample frequency means the proportion of the sample size of each category in the sample data set.

[0025] In one possible implementation, the device further includes:

[0026] an acquisition module, configured to acquire a first sample level of first sample data based on the first sample feature, and to acquire a second sample level of second sample data based on the second sample feature;

[0027] The acquisition module is further used to acquire a first loss weight of the first sample data and a second loss weight of the second sample data;

[0028] a difference determination module, configured to perform weighted processing on the first difference and the second difference based on the first loss weight and the second loss weight, respectively, to obtain a third difference, wherein the first difference is a difference between the first sample level and the first sample label, and the second difference is a difference between the second sample level and the second sample label;

[0029] Correspondingly, the training module is further used to train the first branch network and the second branch network based on the difference between the first similarity and the second similarity, and the third difference, to obtain the target model.

[0030] In one possible implementation, the acquisition module is also used to determine the current iteration number of the current iterative training process; determine the first loss weight based on the ratio between the current iteration number and the total iteration number; and use the difference between the target threshold and the first loss weight as the second loss weight.

[0031] In another aspect, an object classification device is provided, the device comprising:

[0032] A model calling module, configured to call a target model based on any object target image in response to a grading request of any requesting device, wherein the grading request is used to request to return the level of the object corresponding to the target image;

[0033] A returning module, used for returning the level of the object corresponding to the target image to any requesting device;

[0034] Wherein, the target model is trained based on the above-mentioned model training method for object classification.

[0035] On the other hand, a computer device is provided, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0036] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0037] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above method are implemented.

[0038] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0039] The two branch networks are used to sample sample data sets of at least two categories according to different category sampling rates, and the first sample features and the second sample features of the respective sampled data are obtained respectively, so as to provide the possibility for subsequent separation learning through the two branches, and then the two sample features are fused to obtain at least two fused features respectively, and the two sample labels are fused to obtain at least two fused labels corresponding to the at least two fused features, so as to improve the richness of the sample features; the positive sample features and the negative sample features in the at least two fused features are further determined, and the first similarity between the anchor feature and the positive sample feature and the second similarity between the anchor feature and the negative sample feature are determined respectively, so as to further train the network based on the difference between the two similarities to obtain the target model; thus, the network is comparatively learned based on the positive and negative sample pairs, so that the network learns the feature correlation between the object levels in the comparison, so that the two branch networks can better express the common features between adjacent levels and distinguish the differences between non-adjacent interval levels, so that the target model can more accurately perform object classification, thereby improving the accuracy of object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0041] Figure 1 A schematic diagram of an implementation environment for implementing an object classification method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a model training method for object classification provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a module provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of feature fusion provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of comparative learning using features of positive and negative sample pairs provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a model structure for a model training process for object classification provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a flow chart of an object classification method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a model training device for object classification provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the structure of an object grading device provided in an embodiment of the present application;

[0050] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0052] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0053] It will be understood that in the specific implementation of the present application, any user-related data involved, such as the target image, the level of the object corresponding to the target image, the lesion level of the target image, the sample data set, the sample label of the sample data, etc., when the following embodiments of the present application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] The model training method and object classification method for object classification provided in the present application involve artificial intelligence technology, machine learning and other technologies. The big data processing technology in the following artificial intelligence technology can be used to process a large number of sample data sets so as to use the sample data sets for model training. Of course, the above-mentioned machine learning can also be used to train a target model for object classification, so as to use the trained target model for object classification. Exemplarily, the method of the present application can be applied to the field of cloud medical care. For example, the trained target model is mounted on a cloud computing center, and the medical and health service cloud platform created by "cloud computing" technology calls the target model of the cloud computing center to provide users with object classification services. For example, the object classification service may include a service for identifying the level of object lesions based on the target image of the object.

[0056] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0057] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0058] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0059] Medical cloud refers to the use of cloud computing to create a cloud platform for medical and health services based on new technologies such as cloud computing, mobile technology, multimedia, 4G communications, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, hospital appointment registration, medical insurance, etc. are all products of the combination of cloud computing and the medical field. Medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.

[0060] Figure 1 is a schematic diagram of an implementation environment of an object classification method provided by an embodiment of the present invention, see Figure 1 The implementation environment includes: a computer device 101, a computer device 102, and a request device 103. The computer device 102 is configured with a trained target model, and the target model is used to output the level of the object corresponding to the image based on the input image; for example, the level may include a lesion level indicating the severity of the lesion of the object. The computer device 101 provides an object grading service based on the calling process of the target model.

[0061] In one possible example, the requesting device 103 may obtain a target image of any object and send a grading request to the computer device 101, wherein the grading request is used to request the return of the level of the object corresponding to the target image. The computer device 101 receives the grading request, calls the target model in the computer device 102 based on the target image, obtains the level of the object corresponding to the target image, and returns the level of the object corresponding to the target image to the requesting device 103. Exemplarily, the computer device 102 may pre-train an initial model including a first branch network and a second branch network based on a large number of sample data sets to obtain the target model. In another possible example, the computer device 102 may also pre-configure and store the trained target model, and the computer device 102 may directly use the stored target model to return the level of the object corresponding to the target image to the requesting device 103.

[0062] Exemplarily, the computer device 101 and the computer device 102 may be servers. The request device 103 may be configured with a target application, which may be a program plug-in installed in an application platform; for example, the application platform may be a medical and health service cloud platform built based on cloud computing technology, and the object classification service may be one of the multiple services provided by the medical and health service cloud platform; of course, the target application may also be an independent application program, and the specific form of the target application is not limited in the embodiment of the present application.

[0063] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The above-mentioned network may include but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, Wi-Fi and other networks that realize wireless communication. The terminal can be a smart phone, medical cloud equipment, medical self-service terminal, personal computer, tablet computer, laptop computer, digital broadcast receiver, MID (Mobile Internet Devices), PDA (personal digital assistant), desktop computer, vehicle-mounted terminal (such as vehicle-mounted navigation terminal, vehicle-mounted computer, etc.), smart speaker, smart watch, etc. The terminal and the server can be directly or indirectly connected by wired or wireless communication, but it is not limited to this. The specific can also be determined based on the actual application scenario requirements, and it is not limited here.

[0064] Figure 2 The present invention provides a flow chart of a model training method for object classification. The execution subject of the method may be a computer device. Figure 2 As shown, the method includes the following steps.

[0065] Step 201: The computer device samples the sample data set based on the first branch network and extracts a first sample feature of the sampled first sample data, and samples the sample data set based on the second branch network and extracts a second sample feature of the sampled second sample data.

[0066] The sample data set includes samples of at least two categories, and the category refers to the level of the object corresponding to the sample data. For example, the level may include the lesion level of the object. The lesion level is used to indicate the degree of the lesion, and the higher the lesion level, the more severe the corresponding lesion level. For example, the lesion level may include level zero, level one lesion, level two lesion, level three lesion, and level four lesion, wherein level zero indicates no symptoms, level one lesion is a low-level lesion, corresponding to a relatively mild lesion, level two lesions are slightly higher in level, and the corresponding lesion degree is also higher than that of level one lesions, level three lesions are higher in level than level two lesions, and the corresponding lesion degree is also higher than that of level two lesions, and level four lesions are the highest level, and the corresponding lesion degree is the most severe and the most serious. The first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label;

[0067] The initial model to be trained includes a first branch network and a second branch network. The category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of samples of at least two categories with different sample sizes in the sample data set. Exemplarily, the sample sizes of the at least two categories of samples in the sample data set may be different. For example, there are more samples of secondary lesions in the sample data set, and fewer samples of tertiary lesions and quaternary lesions. The category adoption rates of the first branch network and the second branch network for each category may be different.

[0068] In a possible implementation, uniform sampling can be performed through the first branch network, that is, the category sampling rate corresponding to the category with a large sample size is also large; reverse sampling can be performed through the second branch network, that is, the category sampling rate corresponding to the category with a large sample size is smaller, so as to balance the category sampling rate of the category with a small sample size in the first branch network. Then step 201 may include: the computer device can perform uniform sampling on the sample data set based on the first branch network to obtain the first sample data, and extract the first sample feature of the first sample data; the computer device can determine the sample frequency of each category in the sample data set based on the second branch network, and perform reverse sampling on the samples of at least two categories based on the sample frequency of each category to obtain the second sample data, and extract the second sample feature of the second sample data. Among them, the uniform sampling means that the category sampling rate of each category in at least two categories is proportional to the sample frequency of each category, and the sample frequency refers to the proportion of the sample size of each category in the sample data set; the reverse sampling means that the category sampling rate of each category in at least two categories is inversely proportional to the sample frequency of each category.

[0069] Exemplarily, the sample frequencies of level zero, level one lesions, level two lesions, level three lesions, and level four lesions in the sample data set are 20%, 20%, 40%, 10%, and 10%, respectively; then the category sampling rates of the first branch network for level zero, level one lesions, level two lesions, level three lesions, and level four lesions are 20%, 20%, 40%, 10%, and 10%, respectively. For levels with larger sample frequencies, the proportion of the first branch network's sampling volume to the total sampling volume is also larger, which is proportional to the sample frequency; while the category sampling rates of the second branch network for level zero, level one lesions, level two lesions, level three lesions, and level four lesions are 15.4%, 15.4%, 7.6%, 30.8%, and 30.8%, respectively. For levels with larger sample frequencies, the proportion of the second branch network's sampling volume to the total sampling volume is smaller, which is inversely proportional to the sample frequency.

[0070] In one possible example, the first branch network and the second branch network can be sampled by a sampler, and the features of the sampled samples can be extracted by a convolutional network for feature extraction. Exemplarily, the first branch network can uniformly sample the sample data set by a uniform sampler to obtain the first sample data, and extract the first sample feature of the first sample data by a convolutional network. The second branch network can uniformly sample the sample data set by a reverse sampler to obtain the second sample data, and extract the second sample feature of the second sample data by a convolutional network. The structure of the convolutional network of the first branch network and the second branch network can be the same. Exemplarily, in an embodiment of the present application, ResNet-50 (Residual Network) is used as the basic model, and the model structure is shown in Table 1 below. Among them, the stride (step length) of the first layer of Conv3_x (Convolutional, convolutional layer) and Conv4_x is 2. Among them, each convolutional layer is followed by a ReLU (Rectified Linear Unit, linear rectification function, an activation function) layer and a BatchNormalization (batch normalization) layer. As shown in Table 1 below, Table 1 shows the three-dimensional information of ResNet-50, including the layer name, output size, and each convolutional network structure.

[0071] Table 1

[0072]

[0073]

[0074] The sample data set can use 2725 X-ray scan images collected from cooperative hospitals for model training and testing. The data can be divided into training set, validation set and test set in a ratio of 70:10:20. X-Ray penetrating irradiation, the pictures taken can be used in medical treatment.

[0075] The resolution of X-ray scanned images is relatively large, generally 1000×1500. In order to facilitate model processing, the scanned images can be scaled to 352×352, that is, the input of Conv1 in Table 1 is a 352×352 image. Conv1 includes 64 7×7 convolution kernels with a step size of 2. The feature map size of Conv1 output is 176x176. Conv2_x includes 3 modules (blocks), each module includes three convolution layers, the first convolution layer includes 64 7×7 convolution kernels, the second convolution layer includes 64 3×3 convolution kernels, and the third convolution layer includes 256 1×1 convolution kernels. The feature map size of Conv2_x output is 88x88. Conv3_x includes 4 modules (each module includes three convolutional layers as shown in Table 1, and the structure is the same as Conv2_x, but the number has changed, which will not be repeated here), and the feature map size output by Conv3_x is 44x44. Conv4_x includes 6 modules, and the feature map size output by Conv4_x is 22x22. Conv5_x includes 3 modules, and the feature map size output by Conv5_x is 11x11, wherein each feature point in the 11x11 feature map output by Conv5_x includes the features of 2048 channels. Conv5_x is followed by a classifier, which outputs a feature matrix including features of 5 channels, and the features of the 5 channels can correspond to the probabilities of 5 lesion levels. It should be noted that the classifier may include a conversion matrix, through which the 2048-dimensional feature matrix output by Conv5_x is converted into a 5-dimensional feature matrix.

[0076] The structure of each module (block) is as follows Figure 3As shown. The module can be a residual network structure. The input of the module is the 256-dimensional feature matrix output by the previous layer. The first convolution layer in the module includes 64 1×1 convolution kernels, the second convolution layer includes 64 3×3 convolution kernels, and the third convolution layer includes 256 1×1 convolution kernels. Each convolution layer can include a ReLU layer, and can also include a Batch Normalization layer. In each module, after the input feature matrix is ​​processed by each convolution layer, and by the ReLU layer and BatchNormalization layer after each convolution layer, the obtained feature matrix can be added to the 256-dimensional feature matrix input to the module, and the result of the addition is processed by the ReLU layer to obtain the output result of the module.

[0077] It should be noted that the above only uses the structure of ResNet-50 as an example. Of course, other network structures can also be used, such as VGG-16 (Visual Geometry Group) network or DenseNet-121 (Densely Connected Networks) network, etc. The embodiment of the present application is not limited to this.

[0078] Step 202: The computer device fuses the first sample feature and the second sample feature respectively to obtain at least two fused features, and fuses the first sample label and the second sample label to obtain at least two fused labels corresponding to the at least two fused features.

[0079] The computer device may use a fusion parameter to fuse the first sample feature and the second sample feature, and use the same fusion parameter to fuse the sample labels corresponding to the first sample feature and the second sample feature. In one possible implementation, step 202 may include: the computer device obtains at least two fusion parameter pairs, each fusion parameter pair includes two random parameters, and the value range of the two random parameters is greater than 0 and less than 1; for each fusion parameter pair, the computer device fuses the first sample feature and the second sample feature respectively based on the two random parameters included in the fusion parameter pair to obtain the fusion feature corresponding to the fusion parameter pair, and fuses the first sample label and the second sample label to obtain the fusion label of the fusion feature corresponding to the fusion parameter pair. Exemplarily, the computer device may obtain two random numbers between 0 and 1 as a fusion parameter pair.

[0080] Figure 4 A schematic diagram of a fusion process provided by an embodiment of the present application is shown. Figure 4As shown, it is assumed that the sample features generated by the two branches for their respective samples are f c and f r , the corresponding label is y c and r , that is, the true value label of each sample is y c and r , the cube represents f c and f r The corresponding feature matrix, γ is f c The random fusion parameter of f r After fusing the sample features of the two branches based on the two random fusion parameters, the fusion feature f is obtained.

[0081] In one possible example, taking two fusion parameter pairs as an example, the computer device can use the following formula 1 to fuse the first sample feature and the second sample feature to obtain a first fusion feature based on the first fusion parameter pair, and to in-fuse the first sample label and the second sample label to obtain a first fusion label corresponding to the first fusion feature; similarly, based on the second fusion parameter pair, the first sample feature and the second sample feature are fused to obtain a second fusion feature, and the first sample label and the second sample label are in-fused to obtain a second fusion label corresponding to the second fusion feature.

[0082] Formula 1:

[0083] Among them, f c is the first sample feature, f r is the second sample feature, y c is the first sample label corresponding to the first sample feature, that is, the true value label of the first sample data, y r is the second sample label corresponding to the second sample feature, that is, the true value label of the second sample data; γ1 and λ1 are the first fusion parameter pair, f1 is the first fusion feature, and y1 is the first fusion label corresponding to the first fusion feature; γ2 and λ2 are the second fusion parameter pair, f2 is the first fusion feature, and y2 is the first fusion label corresponding to the first fusion feature; wherein, γ1 and λ1, γ2 and λ2 can be random numbers in the range of [0, 1].

[0084] It should be noted that the computer device can take a sample of the first branch network and a sample of the second branch network as a group of samples, and for each group of samples, fuse the sample features of the group of samples and fuse the sample labels, and perform subsequent step 203. In step 202, the computer device can obtain the at least two fusion parameter pairs for each group of samples. The above is only illustrated by two fusion parameter pairs. Of course, if there are three or more fusion parameter pairs, three or more fusion features and their corresponding fusion labels can be obtained based on the process corresponding to the above formula 1 in a similar manner, and they will not be repeated here.

[0085] Step 203: The computer device determines a positive sample feature and a negative sample feature in the at least two fused features based on the anchor point label corresponding to the anchor point feature and the at least two fused labels.

[0086] The anchor feature is any one of the first sample feature and the second sample feature.

[0087] In this step, the computer device determines the anchor feature and the anchor tag corresponding to the anchor feature, and determines the positive sample feature and the negative sample feature in the corresponding at least two fused features based on the similarity between the anchor tag and each fused tag. In one possible implementation, step 203 may include: the computer device determines the similarity between the anchor tag and each fused tag in the at least two fused tags, and obtains at least two similarities; the computer device uses the fused feature corresponding to the third similarity in the at least two similarities as the positive sample feature, and uses the fused feature corresponding to the fourth similarity in the at least two similarities as the negative sample feature, and the third similarity is greater than the fourth similarity. Exemplarily, the computer device can use any one of the first sample feature and the second sample feature as the anchor feature, and the sample tag corresponding to any one of the features is used as the anchor tag corresponding to the anchor feature.

[0088] For example, taking two fusion parameter pairs as an example, we take the first sample feature f c As the anchor, and calculate the anchor label y c The difference between the two fusion labels y1 and y2 respectively, and the fusion feature corresponding to the fusion label with the smaller difference between the two fusion labels y1 and y2 is taken as the positive sample feature f p , the fusion feature corresponding to the fusion label with smaller difference is used as the negative sample feature f n. Of course, if there are three or more fusion features and their corresponding fusion labels, the fusion feature corresponding to the fusion label with the smallest difference among at least two fusion labels can be used as the positive sample feature, and the fusion feature corresponding to the fusion label with the largest difference can be used as the negative sample feature. Alternatively, other screening rules can be used to screen positive and negative sample features. For example, the positive sample feature is the fusion feature corresponding to the fusion label with the smallest difference, and the positive sample feature is the fusion feature corresponding to any fusion label with a difference greater than the minimum difference. The sample label corresponding to the positive sample feature is closer to the anchor label, and the negative sample feature is relatively far away. The embodiment of the present application does not limit the screening rules for the positive and negative sample features in the at least two fusion features.

[0089] Step 204: The computer device determines a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and trains the first branch network and the second branch network based on the difference between the first similarity and the second similarity to obtain a trained target model.

[0090] In a possible implementation, the model training process may include: when the difference between the second similarity and the first similarity is not higher than the target value, the computer device iteratively trains the first branch network and the second branch network until the second similarity is higher than the target value of the first similarity, then stops the training, and obtains the target model. The computer device may repeatedly perform the above steps 201-203, and optimize the model parameters in the first branch network and the second branch network based on the difference between the first similarity and the second similarity in each repetition, so as to iteratively train the initial model, and stop the iteration when the second similarity is higher than the target value of the first similarity, and obtain the target model.

[0091] In a possible example, the computer device may determine the loss value of this iterative training based on the anchor point feature, the positive sample feature, and the negative sample feature through the triple loss function shown in the following formula 2, and stop the iteration when the loss value reaches the first optimization condition to obtain the target model:

[0092] Formula 2:

[0093] in, is the loss value of Triplet loss (triplet loss function), d(x, y) represents the distance between x and y, that is, d(f c ,f p ) is the anchor feature f c And the positive sample feature f p The first distance between them is larger, and the first similarity is smaller; d(fc ,f n ) is the anchor feature f c And negative sample features f n The second distance between the two, the larger the second distance, the smaller the second similarity; for example, the distance can be represented by Manhattan distance (L1 distance), then d(x, y) = ||xy||1. The value of margin (margin parameter) can be the target value.

[0094] It should be noted that when objects are graded, there is continuity between the features of the object levels. For example, when the lesions of an object are graded, the features of each lesion level change continuously, that is, the lesion grading process has the following important grading priors: from primary lesions to secondary lesions and then to tertiary lesions, it is a gradual change process. The severity of secondary lesions increases on the basis of primary lesions, the severity of tertiary lesions increases on the basis of secondary lesions, and the severity of tertiary lesions increases on the basis of tertiary lesions. Correspondingly, there is a continuous correlation between the features of the images corresponding to each level of lesions, that is, the features of the images corresponding to each level of lesions also show a trend of gradual change. Obviously, the feature difference between the secondary lesions and the adjacent primary and tertiary lesions is close or small, while the feature difference between the secondary lesions and the tertiary lesions is relatively large. For example, patients with higher lesion levels (level 3, level 4) often have similar features in the lesions, and this type of representation is obviously different from patients with lower levels (level 1, level 2), that is, the gap between level 4 and level 3 is relatively small, and the gap between level 1 and level 2 is relatively large.

[0095] The embodiment of the present application is based on the above-mentioned grading prior. Through the above-mentioned steps 202-204, the grading prior is embedded in the network learning, the features output by the two branch networks are fused, and the two new lesion features obtained by the fusion are compared and learned. By constructing anchor points and positive and negative sample pairs, it is determined which of the two new lesion features is more serious, and based on this, feature comparison learning is performed on the two branch networks, so that the network learns the correlation between lesion levels in the comparison, so that the target model can more accurately grade lesions, thereby improving the accuracy of lesion grading.

[0096] By fusion of sample features through the above step 202, and corresponding fusion of sample labels, the features of intermediate-level lesions between each level are refined, and the refined expression of features at the lesion level is enhanced. Subsequent optimization of the branch network based on this can further improve the accuracy of the features extracted by the network.

[0097] The positive sample features and negative sample features in the fusion features are determined through the above step 203, and further through step 204, the model is iteratively trained using the difference between the two similarities between the anchor feature and the positive and negative sample features, so that the model can be compared and learned through the positive and negative sample pairs, and the feature expression of the two branch networks is optimized, thereby shortening the distance between the anchor and the positive sample features in the feature space, and at the same time widening the distance between the anchor and the negative sample features. That is, the positive sample features are closer to the anchor features, and the negative sample features are far away from the anchor features in the feature space, so as to achieve the purpose of reducing the feature gap between samples of adjacent levels and increasing the feature gap between samples of non-adjacent interval levels, thereby effectively embedding the important classification of object classification into the network for learning, and improving the accuracy of object classification by the trained target model.

[0098] like Figure 5 As shown in the figure, Triplet loss can shorten the distance between the anchor and the positive (positive sample) in the feature space, and increase the distance between the anchor and the negative (negative sample), so as to reduce the intra-class gap, that is, to reduce the feature difference between adjacent lesion levels; at the same time, it can increase the inter-class gap, that is, to increase the feature difference between non-adjacent interval lesion levels.

[0099] The above process is for learning based on the characteristics of the network. In a possible implementation, the network can also be learned for classification, for example, the network is learned based on the lesion grading results corresponding to the two branch networks. In a possible example, the training process of the target model also includes the following steps S1 to S3.

[0100] Step S1: The computer device obtains a first sample level of first sample data based on the first sample feature, and obtains a second sample level of second sample data based on the second sample feature.

[0101] The computer device can predict a first sample level of the first sample data through a classifier of a first branch network based on the first sample feature; and predict a second sample level of the second sample data through a classifier of a second branch network based on the second sample feature.

[0102] Step S2: The computer device obtains a first loss weight of the first sample data and a second loss weight of the second sample data.

[0103] In one possible implementation, the computer device may determine the current iteration number of the current iterative training process; the computer device determines the first loss weight based on the ratio between the current iteration number and the total iteration number; and the computer device uses the difference between the target threshold and the first loss weight as the second loss weight.

[0104] Exemplarily, the computer device may obtain the first loss weight by the following formula 3:

[0105] Formula 3:

[0106] Among them, α represents the first loss weight, N is the current iteration number of the current iterative training process, and N max is the total number of iterations of iterative training. The target threshold may be 1, and the second loss weight may be (1-α). Of course, the target threshold may be configured based on needs, for example, the target threshold may also be 0.9, 0.95, etc., which is not specifically limited in the embodiments of the present application.

[0107] For example, if the total number of iterations is 30, then during multiple iterations of training, the value of N gradually increases from 1 to 30. Correspondingly, the value of the first loss weight gradually decreases from close to 1 to close to 0, while the value of the second loss weight gradually increases from close to 0 to close to 1.

[0108] Step S3: The computer device performs weighted processing on the first difference and the second difference based on the first loss weight and the second loss weight, respectively, to obtain a third difference.

[0109] The first difference is the difference between the first sample level and the first sample label, and the second difference is the difference between the second sample level and the second sample label. In this step, the computer device obtains the first difference and the second difference, and performs weighted processing on the first difference based on the first loss weight and the second difference based on the second loss weight, respectively, and sums the results obtained by the two weighted processing to obtain the third difference.

[0110] Exemplarily, the computer device may perform weighted processing on the first difference and the second difference based on the first loss weight and the second loss weight by the following formula four corresponding to the loss function to obtain a third difference.

[0111] Formula 4:

[0112] in, represents the loss value of the loss function, which is also the third difference. E(.) is the cross entropy calculation function. Represents the transformation matrix corresponding to the classifier of the first branch network, Represents the transformation matrix corresponding to the classifier of the second branch network, f c is the first sample feature, f r is the second sample feature, correspondingly, represents the first sample level predicted based on the first sample feature of the first branch network, Represents the second sample level predicted based on the first sample feature of the first branch network. c is the first sample label corresponding to the first sample feature, that is, the true value label of the first sample data, y r is the second sample label corresponding to the second sample feature, that is, the true value label of the second sample data. α represents the first loss weight, (1-α) represents the second loss weight, and the values ​​of the two loss weights change with the number of iterations.

[0113] Exemplarily, when the computer device is trained based on the third difference, for example, the parameters in the first branch network and the second branch network can be optimized based on the loss value of the loss function in the above formula 4 to achieve iterative training of the model, until the second optimization condition is reached, the iteration is stopped, and the target model is obtained, wherein the second optimization condition may include but is not limited to: the loss value of the loss function corresponding to the third difference is not higher than 0.05, the loss value is 0, etc. For example, network parameters such as parameters in the convolution layer and the conversion matrix in the classifier can be optimized.

[0114] In one possible implementation, based on the above steps S1 to S3, the computer device trains the first branch network and the second branch network based on the difference between the first similarity and the second similarity to obtain the trained target model. The step can be replaced by: step S4, the computer device trains the first branch network and the second branch network based on the difference between the first similarity and the second similarity, and the third difference to obtain the target model.

[0115] It should be noted that the computer device can perform model training based on the difference between the first similarity and the second similarity, and can perform model training based on the third difference. The model can also be jointly trained based on the difference between the first similarity and the second similarity, and the third difference. The embodiment of the present application does not limit the method of performing joint training based on two differences, or training separately based on two differences. In addition, the embodiment of the present application does not limit the execution order of steps 202-203 and the execution order of steps S1-S3.

[0116] The sample data set can use 2725 X-ray scan images collected from cooperative hospitals for model training and testing. Among them, the data can be divided into training set, validation set and test set in a ratio of 70:10:20. The training set can be used to train the model; the validation set can be used to adjust the hyperparameters of the model and to make a preliminary evaluation of the model's object classification ability. For example, during the iterative training of the model, the current model's generalization ability and accuracy can be verified to refer to whether to stop training. The test set can be used to evaluate the generalization ability and accuracy of the final model. In the testing phase, the test image will be input into two branches at the same time. After obtaining the predicted probability distribution, the two probability distributions will be averaged to obtain the final prediction result.

[0117] The network in the model can be CNN (Convolutional Neural Network), which is a deep learning network widely used in image classification tasks. The network includes convolutional layers, pooling layers, and fully connected layers. Among them, the features that can be output by each convolutional layer in CNN can be in the form of Feature Map. Feature map refers to the feature map obtained by convolution of image and filter. Feature map can be convolved with filter to generate new feature map. Contrastive Learning is to let the machine extract and learn features by inputting constructed positive and negative sample pairs, so that it can better extract commonalities within the domain and differences between domains.

[0118] Taking this level as an example of the lesion level, in the task of lesion grading, there is usually a problem of data imbalance. This is because patients will not feel obvious feelings (such as pain) until they reach a certain lesion level and go to the hospital for treatment. At the same time, due to the existence of regular physical examinations and screenings, the number of normal samples collected by the hospital will also greatly exceed the lesion samples. This leads to two imbalanced relationships in the samples used for model training: on the one hand, the number of normal and lesion samples is unbalanced; on the other hand, there is an imbalance between lesion levels, that is, the number of lesion level samples that cause discomfort to patients is significantly greater than the number of samples at lighter or heavier levels. Using data with an unbalanced sample size between lesion levels to train a model will cause the model prediction to have obvious bias, such as biased towards categories with a large sample size, reducing the generalization of the model. In response to this problem, the applicant proposed the above-mentioned model training method for object grading, which can significantly improve the performance of the trained target model on object grading data with unbalanced samples. By using the training process of the above steps S1-S4, the learning attention can be adaptively switched gradually between the two branch networks through the loss weights of the two branch networks, and the accuracy of the sample data at the level of small sample size can be significantly improved without losing the accuracy of the sample data at the level of large sample size. And on this basis, the process of the above steps 202-204 is further proposed, using the constructed positive and negative sample pairs for contrast learning, so as to embed important classification priors into the network in the object classification task, so that the network learns the common features of continuity between adjacent categories, increases the expression of common features in the domain, and enhances the feature expression of non-adjacent interval categories, that is, the difference between domains, so that the network learns the gradual relationship between object levels in contrast learning, and improves the accuracy of object classification.

[0119] In order to clearly introduce the training process of the embodiment of the present application, Figure 6 The network model structure shown in Figure 1 further explains the above training process. Figure 6 As shown in , the first branch network can be a conventional branch, and the second branch network can be a re-balancing branch. In order to reduce the number of model parameters, the trunk networks of the two branches share weights with each other. Figure 6 As shown in the figure, in the classic branch, uniform sampling is adopted. As shown in the coordinate axis, the category sampling rate gradually decreases from the large category to the small category, where the large category refers to the category with a larger sample size in the sample data set, and the small category refers to the category with a smaller sample size; when uniformly sampling, the probability of each sample being adopted is the same, so the sampling amount of the large category sample is greater than that of the small category; in this classic branch, the first sample data is input into the convolutional network for feature extraction, and after the feature extraction of layers of convolutional layers and the global average pooling processing of the pooling layer, the first sample feature f is obtained. cIn the balanced branch, the reverse sampling method is used to increase the sampling rate from large categories to small categories. In this classic branch, the second sample data obtained by sampling is input into the convolutional network for feature extraction. After the feature extraction of layers of convolutional layers and the global average pooling of the pooling layer, the second sample feature f is obtained. r The two branches are followed by a mixing module and a cumulative learning module. The first sample feature and the second sample feature can be input into the mixing module, and the mixing module is used to perform the above steps 202-204 to calculate the loss value. By comparing positive and negative sample pairs, the network is optimized; and the first sample feature and the second sample feature are input into the accumulation learning module, and the accumulation learning module is used to perform the above steps S1-S4 to calculate the loss value. The network can be optimized through prediction results and true value labels to finally obtain the target model.

[0120] The model training method for object classification provided in the embodiment of the present application samples at least two categories of sample data sets according to different category sampling rates through two branch networks, and obtains the first sample feature and the second sample feature of each sample data respectively, so as to provide the possibility for subsequent separation learning through the two branches, and then fuses the two sample features to obtain at least two fused features respectively, and fuses the two sample labels to obtain at least two fused labels corresponding to the at least two fused features, so as to improve the richness of the sample features; further determine the positive sample features and the negative sample features in the at least two fused features, determine the first similarity between the anchor feature and the positive sample feature, and the second similarity between the anchor feature and the negative sample feature, so as to further train the network based on the difference between the two similarities to obtain the target model; thereby performing comparative learning on the network based on the positive and negative sample pairs, so that the network learns the feature correlation between the object levels in the comparison, so that the two branch networks can better express the common features between adjacent levels, and distinguish the differences between non-adjacent interval levels, so that the target model can more accurately perform object classification, thereby improving the accuracy of object classification.

[0121] Figure 7 The flowchart of an object classification method provided in the embodiment of the present application is shown in FIG. The execution subject of the method may be a computer device. Figure 7 As shown, the method includes the following steps.

[0122] Step 701: The computer device responds to a grading request from any requesting device, calls a target model based on a target image of any object, and obtains the level of the object corresponding to the target image. The grading request is used to request to return the level of the object corresponding to the target image.

[0123] The requesting device may be a user's device, such as a user's terminal. In some possible scenarios, taking the level as the lesion level as an example, a doctor or patient may send a grading request to the computer device based on the lesion grading service provided by the medical platform, and the computer device may call the target model based on the target image of the grading request. Exemplarily, the computer device may be configured with a service interface for the lesion grading service, and based on the service interface, the target image is sent to a device in the cloud computing center that provides the service to obtain the lesion level of the target image.

[0124] Exemplarily, the equipment of the cloud computing center can be configured with a target model trained based on the above-mentioned model training method for object classification, the equipment of the cloud computing center inputs the target image into the target model, extracts the first feature of the target image through the first branch network that has been trained in the target model, and obtains the first probability distribution matrix corresponding to the first feature based on the classifier of the first branch network, the first probability distribution matrix includes the probability of the target image being at each level; for example, including 5 probability values ​​corresponding to five lesion levels; similarly, extracts the second feature of the target image through the second branch network that has been trained in the target model, and obtains the second probability distribution matrix corresponding to the second feature based on the classifier of the second branch network, the second probability distribution matrix includes the probability of the target image being at each level; for example, including 5 probability values ​​corresponding to five lesion levels; the equipment of the cloud computing center sums and averages the first probability distribution matrix and obtains the final probability distribution matrix, and uses the level corresponding to the maximum probability value in the probability distribution matrix as the level of the object corresponding to the target image; for example, uses the lesion level corresponding to the maximum probability value among the 5 probability values ​​as the lesion level of the object corresponding to the target image.

[0125] Step 702: The computer device returns the level of the object corresponding to the target image to any requesting device.

[0126] The target model is trained based on the model training method for object classification provided in the above embodiment. The request device receives the level returned by the computer device and displays the level of the object corresponding to the target image to the user.

[0127] After completing the model training based on the above-mentioned model training method for object classification, the target model is directly mounted on the cloud computing center for easy calling. For example, a user can upload a suspected lesion image through the medical platform and upload it to the medical server of the medical platform, and the medical server calls the target model of the cloud computing center. The server of the cloud computing center can automatically perform a graded evaluation on the uploaded image through the target model, and the server of the cloud computing center returns the evaluation result to the medical server, and the medical server returns the lesion level corresponding to the suspected lesion image to the user.

[0128] The object grading method provided in the embodiment of the present application realizes automatic grading by calling the target model based on a grading request through a computer device, and returning the level of the object corresponding to the target image based on the model calling process; for example, it realizes automatic and accurate assessment of the patient's lesion level, reduces the burden of reading films for clinicians, etc., and the computer device itself does not need to store or train the model, which saves costs, realizes unified calling of the target model by multiple parties, and further improves the convenience and practicality of object grading; and the target model is trained based on the above-mentioned model training method for object grading, which improves the accuracy of object grading.

[0129] Figure 8 A schematic diagram of a model training device for object classification provided in an embodiment of the present application, wherein the initial model to be trained includes a first branch network and a second branch network, such as Figure 8 As shown, the device of this embodiment may include:

[0130] An extraction module 801 is used to sample the sample data set based on the first branch network and extract a first sample feature of the sampled first sample data, and to sample the sample data set based on the second branch network and extract a second sample feature of the sampled second sample data;

[0131] A fusion module 802 is used to fuse the first sample feature and the second sample feature respectively to obtain at least two fused features, and to fuse the first sample label and the second sample label to obtain at least two fused labels corresponding to the at least two fused features;

[0132] A determination module 803 is used to determine a positive sample feature and a negative sample feature in the at least two fused features based on an anchor point label corresponding to the anchor point feature and the at least two fused labels;

[0133] A training module 804 is used to determine a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and based on the difference between the first similarity and the second similarity, train the first branch network and the second branch network to obtain a trained target model;

[0134] The sample data set includes at least two categories of samples, the category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of at least two categories of samples with different sample sizes in the sample data set; the first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label; the anchor feature is any one of the first sample feature and the second sample feature.

[0135] In one possible implementation, the determination module 803 is used to determine the similarity between the anchor point label and each of the at least two fused features, respectively, to obtain at least two similarities; the fused feature corresponding to the third similarity among the at least two similarities is used as the positive sample feature, and the fused feature corresponding to the fourth similarity among the at least two similarities is used as the negative sample feature, and the third similarity is greater than the fourth similarity.

[0136] In one possible implementation, the training module 804 is used to iteratively train the first branch network and the second branch network when the difference between the second similarity and the first similarity is not higher than the target value, and stop training when the second similarity is higher than the target value of the first similarity to obtain the target model.

[0137] In one possible implementation, the fusion module 802 is used to obtain at least two fusion parameter pairs, each fusion parameter pair includes two random parameters, and the value range of the two random parameters is greater than 0 and less than 1; for each fusion parameter pair, based on the two random parameters included in the fusion parameter pair, the first sample feature and the second sample feature are fused respectively to obtain a fusion feature corresponding to the fusion parameter pair, and the first sample label and the second sample label are fused to obtain a fusion label of the fusion feature corresponding to the fusion parameter pair.

[0138] In one possible implementation, the extraction module 801 is used to uniformly sample the sample data set based on the first branch network to obtain the first sample data, and extract the first sample feature of the first sample data; determine the sample frequency of each category in the sample data set based on the second branch network, and based on the sample frequency of each category, reverse sample the samples of the at least two categories to obtain the second sample data, and extract the second sample feature of the second sample data, wherein the reverse sampling means that the sample frequency of at least two categories is inversely proportional to the sampling rate, and the sample frequency means the proportion of the sample size of each category in the sample data set.

[0139] In one possible implementation, the device further includes:

[0140] an acquisition module, configured to acquire a first sample level of the first sample data based on the first sample feature, and to acquire a second sample level of the second sample data based on the second sample feature;

[0141] The acquisition module is further used to acquire a first loss weight of the first sample data and a second loss weight of the second sample data;

[0142] a difference determination module, configured to perform weighted processing on the first difference and the second difference based on the first loss weight and the second loss weight, respectively, to obtain a third difference, wherein the first difference is a difference between the first sample level and the first sample label, and the second difference is a difference between the second sample level and the second sample label;

[0143] Correspondingly, the training module 804 is further used to train the first branch network and the second branch network based on the difference between the first similarity and the second similarity, and the third difference, to obtain the target model.

[0144] In one possible implementation, the acquisition module is also used to determine the current iteration number of the current iterative training process; determine the first loss weight based on the ratio between the current iteration number and the total iteration number; and use the difference between the target threshold and the first loss weight as the second loss weight.

[0145] The model training device for object classification provided in the embodiment of the present application samples at least two categories of sample data sets according to different category sampling rates through two branch networks, and obtains the first sample feature and the second sample feature of each sample data respectively, so as to provide the possibility for subsequent separation learning through the two branches, and then fuses the two sample features to obtain at least two fused features respectively, and fuses the two sample labels to obtain at least two fused labels corresponding to the at least two fused features, so as to improve the richness of the sample features; further determines the positive sample features and the negative sample features in the at least two fused features, determines the first similarity between the anchor feature and the positive sample feature, and the second similarity between the anchor feature and the negative sample feature, so as to further train the network based on the difference between the two similarities to obtain the target model; thereby performing comparative learning on the network based on the positive and negative sample pairs, so that the network learns the feature correlation between the object levels in the comparison, so that the two branch networks can better express the common features between adjacent levels, and distinguish the differences between non-adjacent interval levels, so that the target model can more accurately perform object classification, thereby improving the accuracy of object classification.

[0146] The model training device for object classification of this embodiment can execute the model training method for object classification shown in the above embodiments of this application. The implementation principles are similar and will not be repeated here.

[0147] Fig. 9 A schematic diagram of a structure of an object grading device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device of this embodiment may include:

[0148] The model calling module 901 is used to call the target model based on the target image of any object in response to a grading request of any requesting device, and obtain the level of the object corresponding to the target image, wherein the grading request is used to request to return the level of the object corresponding to the target image;

[0149] A returning module 902 is used to return the level of the object corresponding to the target image to any requesting device;

[0150] The target model is trained based on the above-mentioned model training method for object classification.

[0151] The object grading device provided in the embodiment of the present application realizes automatic grading by calling the target model based on the grading request through the computer device, and returning the level of the object corresponding to the target image based on the model calling process; for example, it realizes automatic and accurate assessment of the patient's lesion level, reduces the burden of clinical doctors in reading films, etc., and the computer device itself does not need to store or train the model, which saves costs, realizes the unified calling of the target model by multiple parties, and further improves the convenience and practicality of object grading; and the target model is trained based on the above-mentioned model training method for object grading, which improves the accuracy of object grading.

[0152] The object grading device of this embodiment can execute the object grading method shown in the above embodiments of the present application. The implementation principle is similar and will not be repeated here.

[0153] Fig.10 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device includes: a memory, a processor and a computer program stored in the memory, and the processor executes the above computer program to implement the model training method for object classification and the steps of the object classification method. Compared with the related art, it can achieve:

[0154] The model training method for object classification provided in the embodiment of the present application samples at least two categories of sample data sets according to different category sampling rates through two branch networks, and obtains the first sample feature and the second sample feature of each sample data respectively, so as to provide the possibility for subsequent separation learning through the two branches, and then fuses the two sample features to obtain at least two fused features respectively, and fuses the two sample labels to obtain at least two fused labels corresponding to the at least two fused features, so as to improve the richness of the sample features; further determine the positive sample features and the negative sample features in the at least two fused features, determine the first similarity between the anchor feature and the positive sample feature, and the second similarity between the anchor feature and the negative sample feature, so as to further train the network based on the difference between the two similarities to obtain the target model; thereby performing comparative learning on the network based on the positive and negative sample pairs, so that the network learns the feature correlation between the object levels in the comparison, so that the two branch networks can better express the common features between adjacent levels, and distinguish the differences between non-adjacent interval levels, so that the target model can more accurately perform object classification, thereby improving the accuracy of object classification.

[0155] The object grading method provided in the embodiment of the present application realizes automatic grading by calling the target model based on a grading request through a computer device, and returning the level of the object corresponding to the target image based on the model calling process; for example, it realizes automatic and accurate assessment of the patient's lesion level, reduces the burden of reading films for clinicians, etc., and the computer device itself does not need to store or train the model, which saves costs, realizes unified calling of the target model by multiple parties, and further improves the convenience and practicality of object grading; and the target model is trained based on the above-mentioned model training method for object grading, which improves the accuracy of object grading.

[0156] In an alternative embodiment, a computer device is provided, such as Fig.10 As shown, Fig.10 The computer device 1000 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the computer device 1000 may also include a transceiver 1004, which may be used for data interaction between the computer device and other computer devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 1004 is not limited to one, and the structure of the computer device 1000 does not constitute a limitation on the embodiments of the present application.

[0157] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0158] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0159] The memory 1003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium\other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0160] The memory 1003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the computer program stored in the memory 1003 to implement the steps shown in the above method embodiment.

[0161] Among them, computer equipment includes but is not limited to: servers, terminals, service clusters, cloud computing centers, etc.

[0162] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0163] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0164] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0165] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0166] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A model training method for object classification, characterized in that: The method is applied to the medical field, the initial model to be trained includes a first branch network and a second branch network, and the method includes: Sampling the sample data set based on the first branch network and extracting a first sample feature of the sampled first sample data, and sampling the sample data set based on the second branch network and extracting a second sample feature of the sampled second sample data; The first sample feature and the second sample feature are respectively fused to obtain at least two fused features, and the first sample label and the second sample label are fused to obtain at least two fused labels corresponding to the at least two fused features; Determine a positive sample feature and a negative sample feature in the at least two fused features based on the anchor point label corresponding to the anchor point feature and the at least two fused labels; Determine a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and train the first branch network and the second branch network based on the difference between the first similarity and the second similarity to obtain a trained target model, so as to perform a graded processing on a target image of any object through the target model to obtain a level of the object corresponding to the target image; The sample data set includes at least two categories of samples, the samples are images of objects, the category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of at least two categories of samples with different sample sizes in the sample data set; the first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label; the anchor feature is any one of the first sample feature and the second sample feature.

2. The method according to claim 1, characterized in that The step of training the first branch network and the second branch network based on the difference between the first similarity and the second similarity to obtain a trained target model includes: When the difference between the second similarity and the first similarity is not higher than the target value, the first branch network and the second branch network are iteratively trained until the second similarity is higher than the target value of the first similarity, and the training is stopped to obtain the target model.

3. The method according to claim 1, characterized in that: The determining, based on the anchor point label corresponding to the anchor point feature and the at least two fusion labels, the positive sample feature and the negative sample feature in the at least two fusion features comprises: Determine the similarity between the anchor tag and each of the at least two fusion tags, respectively, to obtain at least two similarities; The fused feature corresponding to the third similarity among the at least two similarities is used as the positive sample feature, and the fused feature corresponding to the fourth similarity among the at least two similarities is used as the negative sample feature, and the third similarity is greater than the fourth similarity.

4. The method according to claim 1, characterized in that: The fusing the first sample feature and the second sample feature to obtain at least two fused features, and fusing the first sample label and the second sample label to obtain at least two fused labels corresponding to the at least two fused features, include: Obtain at least two fusion parameter pairs, each fusion parameter pair includes two random parameters, and the value range of the two random parameters is greater than 0 and less than 1; For each fusion parameter pair, based on the two random parameters included in the fusion parameter pair, the first sample feature and the second sample feature are fused respectively to obtain the fusion feature corresponding to the fusion parameter pair, and the first sample label and the second sample label are fused to obtain the fusion label of the fusion feature corresponding to the fusion parameter pair.

5. The method according to claim 1, characterized in that The step of sampling the sample data set based on the first branch network and extracting a first sample feature of the sampled first sample data, and the step of sampling the sample data set based on the second branch network and extracting a second sample feature of the sampled second sample data, comprises: Based on the first branch network, uniformly sampling the sample data set to obtain the first sample data, and extracting first sample features of the first sample data, wherein the uniform sampling means that the category sampling rate of each category in at least two categories is proportional to the sample frequency of each category, and the sample frequency means the proportion of the sample size of each category in the sample data set; Based on the second branch network, the sample frequency of each category in the sample data set is determined, and based on the sample frequency of each category, the samples of the at least two categories are reversely sampled to obtain the second sample data, and the second sample features of the second sample data are extracted, wherein the reverse sampling means that the category sampling rate of each category in at least two categories is inversely proportional to the sample frequency of each category.

6. The method according to claim 1, characterized in that The method further comprises: Based on the first sample feature, obtaining a first sample level of the first sample data through the classifier of the first branch network, and based on the second sample feature, obtaining a second sample level of the second sample data through the classifier of the second branch network; Obtain a first loss weight of the first sample data and a second loss weight of the second sample data; Based on the first loss weight and the second loss weight, respectively, weighting the first difference and the second difference to obtain a third difference, wherein the first difference is the difference between the first sample level and the first sample label, and the second difference is the difference between the second sample level and the second sample label; Correspondingly, based on the difference between the first similarity and the second similarity, the first branch network and the second branch network are trained to obtain a trained target model, including: Based on the difference between the first similarity and the second similarity, and the third difference, the first branch network and the second branch network are trained to obtain the target model.

7. The method according to claim 6, characterized in that The obtaining of a first loss weight of the first sample data and a second loss weight of the second sample data comprises: Determine the current iteration number of the current iterative training process; Determining the first loss weight based on a ratio between the current number of iterations and the total number of iterations; The difference between the target threshold and the first loss weight is used as the second loss weight.

8. An object classification method, characterized in that: The method comprises: In response to a grading request of any requesting device, based on a target image of any object, calling a target model to obtain a level of the object corresponding to the target image, wherein the grading request is used to request to return the level of the object corresponding to the target image; Returning the level of the object corresponding to the target image to any requesting device; Wherein, the target model is trained based on the model training method for object classification described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The target image based on any object, calling the target model, and obtaining the level of the object corresponding to the target image, includes: Based on the target image, extract features of the target image through a first branch network of the target model, and obtain a first probability distribution matrix corresponding to the target image based on the extracted first features, wherein the first probability distribution matrix includes the probability of the target image corresponding to each level; Based on the target image, extract features of the target image through a second branch network of the target model, and obtain a second probability distribution matrix corresponding to the target image based on the extracted second features, wherein the second probability distribution matrix includes the probability of the target image corresponding to each level; The first probability distribution matrix and the second probability distribution matrix are averaged to obtain a target probability distribution matrix, and the level corresponding to the maximum probability value in the target probability distribution matrix is ​​determined as the level of the object corresponding to the target image.

10. A model training device for object classification, characterized in that: The device is applied in the medical field, the initial model to be trained includes a first branch network and a second branch network, and the device includes: an extraction module, configured to sample the sample data set based on the first branch network and extract a first sample feature of the sampled first sample data, and to sample the sample data set based on the second branch network and extract a second sample feature of the sampled second sample data; a fusion module, configured to fuse the first sample feature and the second sample feature respectively to obtain at least two fused features, and to fuse the first sample label and the second sample label to obtain at least two fused labels corresponding to the at least two fused features; A determination module, configured to determine a positive sample feature and a negative sample feature in the at least two fused features based on an anchor point label corresponding to the anchor point feature and the at least two fused labels; A training module, used to determine a first similarity between the anchor feature and the positive sample feature, and a second similarity between the anchor feature and the negative sample feature, and based on the difference between the first similarity and the second similarity, train the first branch network and the second branch network to obtain a trained target model, so as to perform a graded processing on a target image of any object through the target model to obtain a level of the object corresponding to the target image; The sample data set includes at least two categories of samples, the samples are images of objects, the category sampling rates of the first branch network and the second branch network are different, and the category sampling rate refers to the sampling rate of at least two categories of samples with different sample sizes in the sample data set; the first sample data corresponds to a first sample label, and the second sample data corresponds to a second sample label; the anchor feature is any one of the first sample feature and the second sample feature.

11. An object classification device, characterized in that: The device comprises: A model calling module, configured to call a target model based on a target image of any object in response to a grading request of any requesting device, wherein the grading request is used to request to return the level of the object corresponding to the target image; A returning module, used for returning the level of the object corresponding to the target image to any requesting device; Wherein, the target model is trained based on the model training method for object classification described in any one of claims 1 to 7.

12. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.