Image classification model training method, image classification method and device

Abstract

The application provides an image classification model training method, an image classification method, an image classification device, electronic equipment and a storage medium, and the method comprises the following steps: combining the first original sample image and the second original sample image to form a training sample set of an image classification model; determining a multi-task loss function of the image classification model according to a use environment of the image classification model; training the image classification model by using the training sample set and the task loss function, determining model parameters of the image classification model, so as to classify a to-be-classified image by using the image classification model, thereby improving the accuracy of image classification model training under the premise of reducing the total amount of training data and without the need of repeatedly performing data labeling, timely and accurately finding an infected person, and saving the time of an epidemiological investigation of an infectious disease.

Description

Technical Field

[0001] This invention relates to disease type information processing technology, and more particularly to image classification model training methods, image classification methods, devices, electronic devices, and storage media. Background Technology

[0002] Currently, people can quickly test for COVID-19 at home by purchasing rapid test strips, kits, and cards. However, this self-testing only completes the biological or chemical testing steps. A problem remains: the reporting and collection of post-test information. Current methods require the tested individual to visually observe and interpret the results before reporting them via mobile phone. Results cannot be automatically and quickly collected and uploaded, failing to fully utilize the advantages of information technology. Furthermore, due to varying levels of proficiency in image uploading and collection, users often fail to upload complete test images, negatively impacting information collection and hindering relevant departments from comprehensively understanding the epidemic situation, thus affecting the prevention and screening of infectious diseases. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image classification model training method, an image classification method, an apparatus, an electronic device, and a storage medium. These methods enable the training of image classification models to stabilize and improve training accuracy through weakly supervised training, while reducing the total amount of training data and eliminating the need for repeated data annotation. This reduces training time and mitigates overfitting defects. Furthermore, the multi-task loss function of the image classification model can flexibly adapt to the detection needs of different diseases, enabling large-scale application of the image classification model. Simultaneously, it allows for accurate identification of images with infection risks based on classification results, enabling timely and accurate detection of infected individuals and saving time in infectious disease epidemiological investigations.

[0004] The technical solution of this invention is implemented as follows: This invention provides an image classification model training method, including: Obtain an unlabeled first raw sample image and a second raw sample image carrying labeled information; The first original sample image and the second original sample image are combined to form a training sample set for the image classification model; Based on the usage environment of the image classification model, determine the multi-task loss function of the image classification model; The image classification model is trained using the training sample set and the task loss function to determine the model parameters, thereby enabling the image to be classified using the image classification model.

[0005] This invention also provides an image classification method, including: Obtain the image to be classified and disease type information, wherein the disease type information includes: Disease type identifier, and the bounding box status of the confirmed image corresponding to the disease type identifier; The training sample set is subjected to feature extraction by the feature extraction network in the image classification model to obtain the image features of the training sample set; The image classification network in the image classification model is used to classify the image features of the training sample set to obtain the classification result of the image to be classified. The image classification model is trained by the method described in any one of claims 1 to 5; When the classification result of the image to be classified is the same as the bounding box state of the confirmed image, a confirmed alarm message is issued.

[0006] This invention also provides an image classification model training device, comprising: The information transmission module is used to acquire an unlabeled first raw sample image and a second raw sample image carrying labeled information; The information processing module is used to combine the first original sample image and the second original sample image to form a training sample set for the image classification model. The information processing module is used to determine the multi-task loss function of the image classification model based on the usage environment of the image classification model. The information processing module is used to train the image classification model using the training sample set and the task loss function, and to determine the model parameters of the image classification model so as to classify the image to be classified using the image classification model.

[0007] In the above scheme, The information processing module is used to determine the disease type corresponding to the image classification model based on the usage environment of the image classification model. The information processing module is used to determine the bounding box type corresponding to the image classification model and the classification loss function corresponding to the bounding box type based on the disease type. The information processing module is used to obtain the confidence loss function of the bounding box; The information processing module is used to calculate the multi-task loss function of the image classification model based on the classification loss function and the confidence loss function.

[0008] In the above scheme, The information processing module is used to perform local augmentation processing on the second original sample image according to the usage environment of the image classification model to obtain a local augmented image, so as to realize spatial filtering on the local neighborhood information of the second original sample image. The information processing module is used to perform global augmentation processing on the second original sample image to obtain a global augmented image, so as to adjust the sharpness of the second original sample image.

[0009] In the above scheme, The information processing module is used to determine a dynamic noise threshold that matches the usage environment of the image classification model. The information processing module is used to denoise the training sample set according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, The information processing module is used to determine a fixed noise threshold corresponding to the image classification model, and to perform denoising processing on the training sample set according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

[0010] In the above scheme, The information processing module is used to adjust the network parameters of the image classification model based on the feature vector of the training sample set and the multi-task loss function. The information processing module is used to determine the network parameters of the image classification model when the loss functions of different dimensions corresponding to the image classification model reach the corresponding convergence conditions, so as to make the parameters of the image classification model compatible with the usage environment.

[0011] This invention also provides an image classification device, comprising: The data transmission module is used to acquire the image to be classified and disease type information, wherein the disease type information includes: Disease type identifier, and the corresponding confirmed image bounding box status; The data processing module is used to extract features from the training sample set through the feature extraction network in the image classification model to obtain the image features of the training sample set; The data processing module is used to classify the image features of the training sample set through the image classification network in the image classification model to obtain the classification result of the image to be classified. The data processing module is used to issue a confirmed alarm message when the classification result of the image to be classified is the same as the bounding box state of the confirmed image.

[0012] This invention also provides an electronic device, the electronic device comprising: Memory, used to store executable instructions; The processor, when running the executable instructions stored in the memory, implements the aforementioned image classification model training method.

[0013] The present invention provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned image classification model training method.

[0014] The embodiments of the present invention have the following beneficial effects: This invention acquires an unlabeled first original sample image and a second original sample image with labeled information; combines the first and second original sample images to form a training sample set for an image classification model; determines the multi-task loss function of the image classification model based on its usage environment; and trains the image classification model using the training sample set and the task loss function to determine its model parameters, thereby enabling the model to classify images to be classified. This allows for training the image classification model with a weakly supervised approach, reducing the total amount of training data and eliminating the need for repeated data labeling, thus steadily improving the accuracy of the image classification model training, reducing training time, mitigating overfitting defects, and allowing the multi-task loss function to flexibly adapt to the detection needs of different diseases, enabling large-scale application of the image classification model. Furthermore, it allows for accurate identification of images with infection risks based on image classification results, timely and accurate detection of infected individuals, and saving time in infectious disease epidemiological investigations. Attached Figure Description

[0015] Figure 1A This is a schematic diagram illustrating the usage environment of the image classification model training method provided in this embodiment of the invention; Figure 1B This is an optional schematic diagram of an image to be classified in an embodiment of the present invention; Figure 2 This is a schematic diagram of the composition structure of the image classification model training device provided in an embodiment of the present invention; Figure 3 This is an optional schematic diagram of an image to be classified in an embodiment of the present invention; Figure 4 This is an optional schematic diagram of an image to be classified in an embodiment of the present invention; Figure 5 A schematic diagram of an optional process for training an image classification model provided in an embodiment of the present invention; Figure 6A This is a schematic diagram illustrating the working principle of the YOLO network in an embodiment of the present invention; Figure 6B This is a schematic diagram of the model structure of the image classification model in an embodiment of the present invention; Figure 7 This is a schematic diagram of the Focus operation in the image classification model in this embodiment of the invention; Figure 8 This is a schematic diagram of a mini-program collecting images to be classified in an embodiment of the present invention; Figure 9 This is a schematic diagram of detection selection in an embodiment of the present invention; Figure 10 This is a schematic diagram of an optional process of the image classification method provided in an embodiment of the present invention.

[0016] Figure 11 This is a schematic diagram of the classification results in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0019] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0020] 1) Based on, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0021] 2) Client: A carrier in a terminal that implements specific functions. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as performing online live streaming (video streaming) or playing online videos.

[0022] 3) Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that include convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. CNNs have representation learning capabilities and can perform shift-invariant classification of input information according to their hierarchical structure.

[0023] 4) Model training: Multi-class classification learning on the image dataset. This model can be built using deep learning frameworks such as TensorFlow and Torch, employing multiple layers of neural networks like CNNs to form a multi-image classification model. The model input is a three-channel or original-channel matrix generated from images read using tools like OpenCV. The model output is the multi-class probability, and the final image classification result is output through algorithms such as softmax. During training, the model approximates the correct trend using objective functions such as cross-entropy.

[0024] 5) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical or computational model in the fields of machine learning and cognitive science that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) and is used to estimate or approximate functions.

[0025] 6) Contrastive loss: This loss function learns a mapping relationship. In high-dimensional space, points of the same class but far apart become closer in low-dimensional space after the mapping, while points of different classes but close together become farther apart in low-dimensional space. The result is that in low-dimensional space, points of the same class will cluster, while the means of different classes will be separated. Similar to Fisher's dimensionality reduction, but Fisher's dimensionality reduction does not have out-of-sample extension and cannot be applied to new samples.

[0026] 7) Soft max: Normalized exponential function, a generalization of the logistic function. It can "compress" a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector, such that each element is in the range [0, 1], and the sum of all elements is 1.

[0027] Figure 1A This is a schematic diagram illustrating a usage scenario of the image classification model training method provided in this embodiment of the invention. (See attached diagram.) Figure 1A The terminals (including terminals 10-1 and 10-2) are equipped with corresponding clients capable of performing different functions. The client (including terminal 10-1) retrieves information from the corresponding server 200 via network 300 to browse different corresponding images, or retrieves corresponding medical images and analyzes the epidemiological survey results of the images. The terminals connect to the server 200 via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission. The types of information from the corresponding images retrieved by the terminals (including terminal 10-1) from the corresponding server 200 via network 300 can be the same or different. For example, the terminals (including terminals 10-1 and 10-2) can retrieve epidemiological survey results matching the images from the corresponding server 200 via network 300, or they can retrieve and browse epidemiological surveys of populations (e.g., close contacts, secondary close contacts, and concurrent cases) related to the current target from the corresponding server 200 via network 300. Server 200 can store information about the corresponding images for different images, and can also store epidemiological surveys matching the information of the corresponding images. In some embodiments of the present invention, the disease type information stored in server 200 may include: information on various types of infectious diseases, each type of infectious disease can be distinguished by a corresponding disease type identifier, and can also store a confirmed diagnosis warning threshold corresponding to the disease type identifier. After obtaining the classification result of the image to be classified through the image classification model, the confirmed diagnosis warning threshold is used to promptly issue a confirmed diagnosis alarm to notify the relevant disease control department. The disease type identifier carried by the disease information in this application can characterize various types of infectious diseases; specifically, infectious diseases are divided into Class A, Class B, and Class C. Taking type A coronavirus infection as an example, terminal 10-2 can perform self-testing using an antigen test strip to determine whether it is diagnosed with type A coronavirus infection, referencing... Figure 1B , Figure 1B This is an optional schematic diagram of the image to be classified in an embodiment of the present invention. In this diagram, both the T-line and the C-line show clearly visible red bands, indicating a positive result; the darker the T-line, the stronger the positive result. Only the C-line shows a clearly visible red band, indicating a negative result. If the C-line does not show a red band, the image is considered invalid. Data is collected via terminal 10-2. Figure 1B The image shown allows server 200 to obtain the corresponding classified image and perform image classification processing using the image classification model deployed in server 200 to determine whether the user is infected with coronavirus A.

[0028] In this invention, embodiments can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. The backend services of network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites; therefore, cloud technology needs cloud computing as its support.

[0029] It's important to note that cloud computing is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, and can be used on demand, expanded at any time, and paid for based on usage. As the foundational providers of cloud computing capabilities, they establish cloud resource pool platforms, often referred to as cloud platforms or Infrastructure as a Service (IaaS). These platforms deploy various types of virtual resources within the resource pool for external customers to choose from. The cloud resource pool primarily includes: computing devices (which can be virtualized machines containing operating systems), storage devices, and network devices.

[0030] In conjunction with the embodiments Figure 1A As shown, the image classification method provided in this embodiment of the invention can be implemented through corresponding cloud devices. For example, terminals (including terminals 10-1 and 10-2) connect to a server 200 located in the cloud via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. It is worth noting that the server 200 can be a physical device or a virtualized device.

[0031] Specifically, in conjunction with the preceding embodiments Figure 1A As shown, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0032] It should be noted that, regardless of the existing infectious disease, the image classification model can be trained based on the image classification model training method of this embodiment during epidemiological investigations. This facilitates remote review and use by doctors. The image classification model can also be fine-tuned according to the type of disease to meet the needs of epidemiological investigations. For example, when the coronavirus A test strip is used as the image to be classified, it includes: a sample addition window, a detection window, and a marker line window (C line and T line). When the HIV test strip is used as the image to be classified, it includes: a sample addition window and a result judgment window.

[0033] Server 200 sends information about the corresponding image of the same image to terminals (terminal 10-1 and / or terminal 10-2) via network 300 to enable users of terminals (terminal 10-1 and / or terminal 10-2) to analyze the information about the corresponding image. As an example, server 200 deploys a corresponding neural network model. Before deployment, the image classification model needs to be trained, specifically including: acquiring an unlabeled first original sample image and a second original sample image carrying labeled information; combining the first and second original sample images to form a training sample set for the image classification model; determining the multi-task loss function of the image classification model based on its usage environment; and training the image classification model using the training sample set and the task loss function to determine the model parameters, thereby enabling the image to be classified using the image classification model.

[0034] The structure of the image classification model training device according to an embodiment of the present invention will be described in detail below. The image classification model training device can be implemented in various forms, such as a dedicated terminal with image classification model training device processing function, or a server with image classification model training device processing function, for example, the preceding... Figure 1A Server 200. Figure 2 This is a schematic diagram of the composition structure of the image classification model training device provided in an embodiment of the present invention. It can be understood that... Figure 2 This is only an exemplary structure of the image classification model training device, not the entire structure; it can be implemented as needed. Figure 2 The structure shown may be part or all of the structure.

[0035] The image classification model training apparatus provided in this embodiment of the invention includes: at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the image classification model training apparatus are coupled together via a bus system 205. It can be understood that the bus system 205 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 205.

[0036] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0037] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of a terminal (such as 10-1). Examples of this data include any computer programs used to operate on the terminal (such as 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0038] In some embodiments, the image classification model training device provided in this invention can be implemented using a combination of hardware and software. For example, the image classification model training device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the image classification model training method provided in this invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0039] As an example of the image classification model training device provided in this embodiment of the invention, which adopts a combination of hardware and software, the image classification model training device provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus 205) to complete the image classification model training method provided in this embodiment of the invention.

[0040] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0041] As an example of the hardware implementation of the image classification model training device provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the image classification model training method provided in this embodiment of the invention.

[0042] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the image classification model training device. Examples of such data include: any executable instructions for operation on the image classification model training device, such as executable instructions that can be included in the program implementing the image classification model training method of this embodiment of the invention.

[0043] In other embodiments, the image classification model training device provided in this invention can be implemented in software. Figure 2An image classification model training device stored in memory 202 is shown. This device can be software in the form of programs and plugins, and includes a series of modules. As an example of a program stored in memory 202, it may include the image classification model training device. The image classification model training device includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the image classification model training device are read into RAM and executed by processor 201, the image classification model training method provided in this embodiment of the invention will be implemented. The functions of each software module in the image classification model training device include: The information transmission module 2081 is used to acquire an unlabeled first original sample image and a second original sample image carrying labeled information.

[0044] The information processing module 2082 is used to combine the first original sample image and the second original sample image to form a training sample set for the image classification model.

[0045] The information processing module 2082 is used to determine the multi-task loss function of the image classification model based on the usage environment of the image classification model.

[0046] The information processing module 2082 is used to train the image classification model using the training sample set and the task loss function, and to determine the model parameters of the image classification model so as to classify the image to be classified using the image classification model.

[0047] Once the image classification model has been trained, image classification devices can be further deployed in electronic devices, including: The data transmission module is used to acquire the image to be classified and disease type information, wherein the disease type information includes: a disease type identifier and the bounding box state of the confirmed image corresponding to the disease type identifier; the data processing module is used to extract features from the training sample set through the feature extraction network in the image classification model to obtain the image features of the training sample set; the data processing module is used to classify the image features of the training sample set through the image classification network in the image classification model to obtain the classification result of the image to be classified; the data processing module is used to issue a confirmed alarm message when the classification result of the image to be classified is the same as the bounding box state of the confirmed image.

[0048] Continue to combine Figure 1AThe illustrated use case explains the image classification model training method provided in this application. Firstly, it describes the image processing procedure for disease diagnosis, using coronavirus A infection as an example. Terminal 10-2 can perform self-testing using antigen test strips to determine whether it is diagnosed with coronavirus A infection. (Refer to...) Figure 3 and Figure 4 , Figure 3 This is an optional schematic diagram of an image to be classified in an embodiment of the present invention. Figure 4 This is an optional schematic diagram of the image to be classified in an embodiment of the present invention. The results of the type A coronavirus antigen test can be divided into three categories: positive, negative, and invalid. If both the C-line and T-line are present, it is considered positive; if only the C-line is present, it is considered negative; and if the C-line is absent, it is considered invalid. However, in practical applications, it is difficult to obtain satisfactory judgment results directly using only this rule. Figure 3 As shown, due to product defects, it is not possible to determine the presence of C and T characters and marking lines solely by comparison, because the C and T characters on some antigen reagents are not very obvious, which is not conducive to automatic identification; another type is... Figure 4 As shown, users unfamiliar with information collection often cannot obtain complete images to be classified, thus affecting the accuracy of recognition. Figure 4 When close contacts of the images to be classified, as shown, develop symptoms after the quarantine period, traditional isolation and control measures for close contacts become ineffective to some extent, posing a risk of further disease transmission. Manual data collection cannot keep pace with the speed of infectious disease transmission and increases the workload of repetitive manual statistics.

[0049] To address the aforementioned deficiencies, refer to Figure 5 , Figure 5 This is an optional flowchart illustrating the image classification model training method provided in this embodiment of the invention, wherein the images can be selected for different disease type information prediction scenarios. It is understood that... Figure 5 The steps shown can be performed by various electronic devices that run image classification model training devices, such as dedicated terminals, servers, or server clusters with image classification processing capabilities. The following section addresses... Figure 5 The steps shown are explained.

[0050] Step 501: The image classification model training device acquires an unlabeled first original sample image and a second original sample image carrying labeled information.

[0051] In some embodiments of the present invention, taking coronavirus A infection as an example, terminal 10-2 can perform self-testing using antigen test strips to determine whether coronavirus A infection has been diagnosed, and the generated image to be classified is as follows: Figure 1BAs shown, during the training phase of the image classification model, it is necessary to add windows, detection windows, and labeling windows for the C-line and T-line in the original sample images, which increases the annotation cost and reduces the available data. To increase the amount of sample image data and improve the training accuracy of the classification model, a weakly supervised loss based on overall image annotation is introduced. The original sample images used for weakly supervised loss can be divided into two types: one is the type A coronavirus antigen detection image used as the second original sample image (positive or negative can be used), and the other is an irrelevant or invalid image used as the first original sample image.

[0052] In some embodiments of the present invention, during the initial stage of training sample acquisition, the clarity of the second original sample image may be affected due to the user's different image acquisition devices and unfamiliarity with the acquisition method. To ensure the clarity of the second original sample image, the following processing can be performed: Based on the usage environment of the image classification model, local augmentation processing is performed on the second original sample image to obtain a locally augmented image, thereby performing spatial filtering on the local neighborhood information of the second original sample image; global augmentation processing is also performed on the second original sample image to obtain a globally augmented image, thereby adjusting the sharpness of the second original sample image. The second original sample image is denoted as X, and two types of transformations are performed: global augmentation and local augmentation. Global augmentation can derive a transformation function that maps input colors to output colors. Local augmentation can perform spatial filtering based on local neighborhood information to determine pixel colors. In some embodiments of the present invention, to enhance the sample processing effect, during image processing, only a local transformation is performed for the enhancement in the X1 direction of the local augmentation, while both global and local transformations are performed simultaneously on the X2 side of the global augmentation result, enabling the image classification model to better learn the relationship between the global augmentation and the local augmentation. It should be noted that examples of data augmentation operations include, but are not limited to, scaling, color dithering, and Gaussian filtering, etc., and this application does not impose specific limitations on these.

[0053] Step 502: The image classification model training device combines the first original sample image and the second original sample image to form a training sample set for the image classification model.

[0054] In some embodiments of the present invention, a dynamic noise threshold is determined that matches the usage environment of the image classification model; The initial training sample set is denoised according to the dynamic noise threshold to form an initial training sample set that matches the dynamic noise threshold. Since the image classification model is used in different environments, the dynamic noise threshold matching the environment of the image classification model is also different. For example, taking coronavirus A infection as an example, the large difference in the proportion of positive, negative, and invalid sample images will affect the training speed of the image classification model. Therefore, the proportion of positive sample images (i.e., the second original sample images) in the training sample set can be increased. Similarly, for the HIV testing environment, the proportion of negative sample images (i.e., the first original sample images) in the training sample set can be increased according to the dynamic noise value to improve the training accuracy of the image classification model.

[0055] In some embodiments of the present invention, a fixed noise threshold corresponding to the image classification model can be determined, and the initial training sample set can be denoised according to the fixed noise threshold to form an initial training sample set that matches the fixed noise threshold. When the image classification model is embedded in a corresponding hardware device, such as a handheld detection terminal, and the environment is for detecting coronavirus A infection, the noise is relatively uniform. By fixing the fixed noise threshold corresponding to the image classification model, the training speed of the image classification model can be effectively improved, the user's waiting time can be reduced, and the large-scale use of the image classification model in coronavirus A infection detection can be facilitated.

[0056] Step 503: The image classification model training device determines the multi-task loss function of the image classification model based on the usage environment of the image classification model.

[0057] In some embodiments of the present invention, the multi-task loss function of the image classification model is determined according to the usage environment of the image classification model, which can be achieved in the following ways: Based on the usage environment of the image classification model, the disease type corresponding to the image classification model is determined; based on the disease type, the bounding box type corresponding to the image classification model and the classification loss function corresponding to the bounding box type are determined; the confidence loss function of the bounding box is obtained; based on the classification loss function and the confidence loss function, the multi-task loss function of the image classification model is calculated. Taking coronavirus A infection as an example, terminal 10-2 can perform self-testing using antigen test strips to determine whether it is diagnosed with coronavirus A infection. The generated image to be classified is as follows: Figure 1B As shown, the image classification model can include the YOLO network (You Only Look Once), see reference. Figure 6A , Figure 6AThis is a schematic diagram illustrating the working principle of the YOLO network in this embodiment of the invention. Utilizing the YOLO network, which is part of the image classification model, three different regions in the corresponding image to be tested on the antigen test strip can be identified: the sample addition window, the detection window, and the marker line window (including the T line and the C line). By analyzing the relative positional relationship of these regions in the image to be tested, the image can be classified. A clear red band on both the T line and the C control line indicates a positive result; the darker the T line, the stronger the positive result. A clear red band on only the C control line indicates a negative result. If no red band appears on the C control line, the result is invalid.

[0058] When the YOLO network is working, the image classification model first divides the input sample image into S... The network uses S grid cells, each responsible for detecting targets whose center point falls within that cell. For each cell, the network predicts B bounding boxes. The prediction output includes the confidence score that the bounding box is a target, the coordinates of the bounding box, and the probability of each bounding box across multiple classes. Therefore, for each image, the network ultimately outputs S... S (5 There are B+C) predicted values, where C represents the number of probability values ​​for each category, and for each grid cell (the image has a total of S predicted values). X grids), 5 B can be represented as two parts: 4 B+1 B, of which 4 B represents the number of coordinates for the top-left and bottom-right corners of the detection box, 1 B represents the detection box confidence. Ultimately, the YOLO network can be used to add windows to samples in the image to be detected, and to identify detection windows and marker line windows (including T lines and C lines).

[0059] Since the test strip image for coronavirus A has three test frames, C=3 here. Figure 6AAs shown, if an image falls within a square, the first parameter is set to 1; otherwise, it is set to 0. Each bounding box has five parameters to predict: x, y, h, w, and confidence. (x, y) represents the coordinates of the box's center point, related to the square's boundaries. (h, w) represents the box's width and height, related to the entire image. Windows with confidence levels below the threshold are then removed based on a threshold. Finally, non-maximum suppression (NMS) is used to remove redundant windows to obtain the final prediction result. NMS is commonly used in computer vision tasks to suppress detection results that are not maxima; here, it mainly refers to NMS in object detection tasks, removing redundant overlapping detection boxes. The detection results are sorted by category in descending order of probability, specifically including the following steps: (a) Assume that the bounding boxes predicted as detection boxes in the image classification detection task are A, B, C, D, and E in descending order; (b) Select the box with the highest probability, A, and mark it as the accept box. Determine the IoU value (the overlapping part of the two regions divided by the sum of the two regions) between the remaining boxes B, C, D, and E and A; (c) Generally, the IoU threshold is set to 0.2~0.5. Boxes greater than this threshold are considered redundant and need to be discarded. Assume that B exceeds the threshold and is discarded, while C, D, and E do not exceed the threshold; (d) Continue from the remaining detection boxes, select the box with the highest probability, C, and mark it as the accept box. Calculate the IoU value between boxes D, E, and C. Again, boxes with an IoU greater than the threshold are discarded; (e) Repeat the above process iteratively until all detection boxes of each category have been processed. Through non-maximum suppression processing, it is possible to detect each detection box in the classified image in real time.

[0060] Step 504: The image classification model training device trains the image classification model using the training sample set and the task loss function to determine the model parameters of the image classification model, so as to classify the image to be classified using the image classification model.

[0061] In some embodiments of the present invention, the image classification model is trained using the training sample set and the task loss function to determine the model parameters of the image classification model, which can be achieved in the following ways: Based on the feature vectors of the training sample set and the multi-task loss function, the network parameters of the image classification model are adjusted until the loss functions of different dimensions corresponding to the image classification model reach the corresponding convergence conditions. Then, the network parameters of the image classification model are determined to ensure that the parameters of the image classification model are adapted to the usage environment. To better illustrate the construction process of the multi-task loss function and the training process of the image classification model, the following explanation uses an image classification model employing the YOLOv5 architecture as an example to illustrate the training process.

[0062] refer to Figure 6B , Figure 6B This is a schematic diagram of the image classification model structure in an embodiment of the present invention. The YOLOv5 structure mainly consists of four parts: the input end, the feature extraction network (Backbone), the Neck structure, and the Prediction structure. The Neck and Prediction structures can constitute the image classification network in the image classification model. First, Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling are performed at the input end. Data augmentation is applied at the input end by randomly scaling, cropping, and rearranging the original image. This enriches the image background and indirectly increases the batch size (the number of samples selected in one training iteration), reducing the training's dependence on the batch size itself.

[0063] Secondly, focus slicing and the CSPDarknet53 structure were added to the backbone. Darknet53 is the backbone network of YOLOv3, which is an improvement on ResNet. While retaining ResNet's ability to represent features, Darknet53 avoids the gradient vanishing problem caused by the excessive depth of ResNet. Darknet53's accuracy on ImageNet is comparable to that of ResNet-152, and its speed is much faster.

[0064] The CSP structure divides the input into two parts: one part is calculated using a residual block, and the other part is directly processed using a short cut. Finally, a concat operation is performed to concatenate the two parts. For example... Figure 3 As shown, a short-cut operation is employed. The CSP structure can reduce some computational load and greatly enrich gradient combinations. This reduction in computational load also means improved inference speed. By avoiding redundant gradient information and increasing the difference in gradient information through the Concat operation, the learning ability of the CNN (Convolutional Neural Network) is enhanced, which also means improved inference accuracy.

[0065] The CSPDarknet53 architecture combines the CSPNet and Darknet53 networks, offering advantages such as low computational cost and high accuracy. (Reference) Figure 7 , Figure 7 This is a schematic diagram of the Focus operation in the image classification model of this invention embodiment, as shown below. Figure 7As shown, the Focus operation before entering the Backbone transforms a 608×608×3 RGB three-channel color image into a 304×304×12 feature map, reducing the input size and quadrupling the number of channels to decrease floating-point computation and improve speed. Further, after a convolutional (CBL) operation, it becomes a 304×304×64 feature map. The Focus layer transforms the information from the wh plane to the channel dimension, and then... 3. Convolutional methods are used to extract different features. This approach reduces information loss caused by downsampling, thereby improving the training accuracy of the image classification model and ensuring accurate identification of coronavirus A test strip images.

[0066] The Neck region plays a crucial role in the network, fusing and extracting features. It employs an FPN+PAN structure, resulting from the fusion of an FPN (Feature Pyramid Network) and a PANet (Path Aggregation Network). Figure 5 As shown, FPN operates from top to bottom, fusing top-level features with bottom-level features through upsampling and concat operations. The Bottom-up Path Augmentation part, in contrast to FPN, employs a bottom-up feature pyramid structure. This better integrates parameters between the bottom and top layers, improving detection performance. The bottom-up operation (left arrow) requires hundreds of network layers, resulting in significant information loss in shallow layers; while the network layer traversed (right arrow) is shallower, better preserving shallow information, reducing information loss, and enhancing the network's ability to extract and fuse features.

[0067] In the prediction section, the most important part is the definition of the loss function, which serves as the criterion for the next backpropagation. The prediction part uses the Generalized Intersection over Union (CIOU) loss to calculate the bounding box loss. The loss function in the prediction part includes the classification loss and confidence loss of the bounding box, combined with... Figure 1B Taking coronavirus A infection as an example, we can use A to represent the predicted bounding box, B to represent the true bounding box, and C to represent the smallest closed box containing both A and B. The Intersection over Union (IOU) calculation formula is shown in Formula 1.

[0068] Formula 1 The classification loss consists of two parts: the first is the confidence loss of the bounding box, representing whether the box is the correct target box; the second is the classification loss, representing which class the bounding box should belong to. Both are calculated using the cross-entropy loss function, with reference to Formula 2: Formula 2 Where N represents the total number of bounding boxes and M represents the number of categories. For the confidence loss in this invention, M=2, while for the classification loss, M=3. This represents the true class of the sample; when sample i belongs to class c... =1, otherwise 0. This represents the probability that sample i belongs to class c, as predicted by the network.

[0069] The image classification model processes the data to obtain the final SxSx(5) After generating B+C) predicted values, we can take C channel results, i.e., SxSxC outputs. Then, we perform global max pooling on these outputs to obtain C outputs. This represents the probability of a sample addition window, detection window, C-line, and T-line existing in the image, using the maximum probability of all possible bounding boxes. For images detecting coronavirus A, these boxes should exist regardless of whether the image is positive or negative; however, for irrelevant images, these boxes should not exist. At this point, the image classification model has completed training and can be deployed on a suitable server or cloud server network.

[0070] refer to Figure 8 , Figure 8 This is a schematic diagram illustrating the process of a mini-program collecting images to be classified in an embodiment of the present invention. Users of instant messaging clients can collect and upload images to be classified through the mini-program on their instant messaging client. It should be noted that, to ensure detection accuracy, when using the image classification model provided in this application, only images to be classified can be collected in real time; images stored locally on the terminal cannot be accessed. When a user... Figure 8 When the mini-program shown collects images to be classified, Figure 9 This is a schematic diagram of detection selection in an embodiment of the present invention, which can be shown to the user. Figure 9Taking three infectious diseases—virus C, coronavirus A, and virus B—as examples, all three are transmitted through contact and are acute infectious diseases. However, virus C and coronavirus A are respiratory infectious diseases. When the disease type is determined to be a respiratory infectious disease, the knowledge graph identifies the first mode of contact as airborne, the first transmission medium as droplets, dust, and aerosols, and the first susceptible population as those exposed to droplets, dust, and aerosols. Virus B is a digestive tract infectious disease. When the disease type is determined to be a digestive tract infectious disease, the knowledge graph identifies the second mode of contact as fecal-oral transmission, the second transmission medium as the surrounding environment, food, and water contaminated with feces, and the second susceptible population as those exposed to fecal-contaminated environments, food, and water. Users can flexibly choose the type of disease to be tested according to their needs.

[0071] refer to Figure 10 , Figure 10 This is an optional process diagram of the image classification method provided in an embodiment of the present invention, which specifically includes the following steps: Step 1001: Obtain the image to be classified and disease type information, wherein the disease type information includes: disease type identifier and the bounding box status of the confirmed image corresponding to the disease type identifier.

[0072] Step 1002: Extract features from the training sample set using the feature extraction network in the image classification model to obtain the image features of the training sample set.

[0073] Step 1003: Classify the image features of the training sample set using the image classification network in the image classification model to obtain the classification result of the image to be classified.

[0074] The classification results of the image to be classified are as follows: Figure 11 As shown, the first step is to determine the completeness of these detection areas. If any of the three categories is missing, the user will be prompted that the uploaded result is invalid. If all three categories of boxes exist and there are two marker lines, the patient will be classified as positive. If only one marker line is detected, as mentioned earlier, the C and T characters are not obvious. The relative positions of the sample addition window and the detection window are compared to determine whether the marker line is a C or a T. Because the orientation of the user-uploaded image can vary, it cannot be confirmed that the C line is always above the T line. However, according to the current design of the type A coronavirus antigen test kit, the sample addition window is always below the detection window. Therefore, we consider that if the marker line is in the detection window and is closer to the half of the sample addition window, it is a T line; otherwise, it is a C line. The output results are described as follows:

[0075] Step 1004: When the classification result of the image to be classified is the same as the bounding box state of the confirmed image, a confirmed alarm message is issued.

[0076] Taking coronavirus A as an example, when the classification result of an image to be classified is confirmed as a positive case, the user corresponding to the image may experience symptoms such as muscle weakness, sensory symptoms, aphasia, blurred vision, dizziness, headache, nausea, vomiting, cognitive impairment, and altered consciousness. This indicates that the user may have been infected with coronavirus A, thus requiring timely issuance of a confirmation alert. The image to be classified is collected through a mini-program on an instant messaging client and sent to a cloud server. When the classification result of the image to be classified matches the bounding box state of the confirmed image, the registered user information of the instant messaging client is sent to the cloud server. This enables the tracking of the target object corresponding to the image to be classified through the cloud server, thereby facilitating the isolation and treatment of the user corresponding to the image.

[0077] Beneficial technical effects: This invention acquires an unlabeled first original sample image and a second original sample image with labeled information; combines the first and second original sample images to form a training sample set for an image classification model; determines the multi-task loss function of the image classification model based on its usage environment; and trains the image classification model using the training sample set and the task loss function to determine its model parameters, thereby enabling the model to classify images to be classified. This allows for training the image classification model with a weakly supervised approach, reducing the total amount of training data and eliminating the need for repeated data labeling, thus steadily improving the accuracy of the image classification model training, reducing training time, mitigating overfitting defects, and allowing the multi-task loss function to flexibly adapt to the detection needs of different diseases, enabling large-scale application of the image classification model. Furthermore, it allows for accurate identification of images with infection risks based on image classification results, timely and accurate detection of infected individuals, and saving time in infectious disease epidemiological investigations.

[0078] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training an image classification model, characterized in that, The method includes: Acquire a first unlabeled original sample image and a second original sample image carrying labeling information; wherein, the first original sample image is an irrelevant image, and the second original sample image is an image for detecting coronavirus A antigen; The first original sample image and the second original sample image are combined to form a training sample set for the image classification model; Based on the usage environment of the image classification model, the multi-task loss function of the image classification model is determined. This multi-task loss function includes the bounding box confidence loss and the classification loss, both calculated based on the following cross-entropy loss function: ; Where N represents the total number of bounding boxes and M represents the number of categories; the confidence loss represents whether the bounding box is a correct target box, and for the confidence loss, M=2; the classification loss represents which category the bounding box belongs to, and for the classification loss, M=3, the corresponding categories are sample addition window, detection window and label line window; This represents the true class of the sample; when sample i belongs to class c... =1, otherwise 0; This represents the probability that sample i belongs to class c, as predicted by the network. The image classification model is trained using the training sample set and the multi-task loss function. The image classification model uses the maximum probability of all possible detection boxes in the image to represent the probability of the presence of the sample addition window, the detection window, the C-line, and the T-line in the image. For positive and negative images of the coronavirus A antigen detection, the sample addition window, the detection window, and the marker line window all exist; for irrelevant images, none of these three windows exist. The model parameters of the image classification model are determined so that the image to be classified can be classified using the image classification model.

2. The method according to claim 1, characterized in that, The step of determining the multi-task loss function of the image classification model based on its usage environment includes: Based on the usage environment of the image classification model, determine the disease type corresponding to the image classification model; Based on the disease type, determine the bounding box type corresponding to the image classification model, and the classification loss function corresponding to the bounding box type; Obtain the confidence loss function of the bounding box; The multi-task loss function of the image classification model is calculated based on the classification loss function and the confidence loss function.

3. The method according to claim 1, characterized in that, The method further includes: Based on the usage environment of the image classification model, the second original sample image is subjected to local augmentation processing to obtain a local augmented image, so as to realize spatial filtering of the local neighborhood information of the second original sample image; The second original sample image is subjected to global augmentation processing to obtain a global augmented image, thereby adjusting the sharpness of the second original sample image.

4. The method according to claim 1, characterized in that, The method further includes: Determine a dynamic noise threshold that matches the usage environment of the image classification model; The training sample set is denoised according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, A fixed noise threshold corresponding to the image classification model is determined, and the training sample set is denoised according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

5. The method according to claim 1, characterized in that, The step of training the image classification model using the training sample set and the multi-task loss function to determine the model parameters of the image classification model includes: Based on the feature vectors of the training sample set and the multi-task loss function, adjust the network parameters of the image classification model; The network parameters of the image classification model are determined when the loss functions of different dimensions corresponding to the image classification model reach the corresponding convergence conditions, so as to make the parameters of the image classification model compatible with the usage environment.

6. An image classification method, characterized in that, The method includes: Obtain the image to be classified and disease type information, wherein the disease type information includes: Disease type identifier, and the bounding box status of the confirmed image corresponding to the disease type identifier; The image features of the training sample set are obtained by using the feature extraction network in the image classification model to extract features from the training sample set. The image classification network in the image classification model is used to classify the image features of the training sample set to obtain the classification result of the image to be classified. The image classification model is trained by the method described in any one of claims 1 to 5; When the classification result of the image to be classified is the same as the bounding box state of the confirmed image, a confirmed alarm message is issued.

7. The method according to claim 6, characterized in that, The method further includes: Images to be classified are collected through a mini-program on an instant messaging client and sent to a cloud server. When the classification result of the image to be classified is the same as the bounding box state of the confirmed image, the registered user information of the instant messaging client is sent to the cloud server so as to realize the tracking of the target object corresponding to the image to be classified through the cloud server.

8. An image classification model training device, characterized in that, The device includes: The information transmission module is used to acquire an unlabeled first original sample image and a second original sample image carrying labeling information; wherein, the first original sample image is an irrelevant image, and the second original sample image is a coronavirus A antigen detection image; The information processing module is used to combine the first original sample image and the second original sample image to form a training sample set for the image classification model. The information processing module is used to determine the multi-task loss function of the image classification model according to the usage environment of the image classification model. The multi-task loss function includes the bounding box confidence loss and the classification loss, both calculated based on the following cross-entropy loss function: ; Where N represents the total number of bounding boxes and M represents the number of categories; the confidence loss represents whether the bounding box is a correct target box, and for the confidence loss, M=2; the classification loss represents which category the bounding box belongs to, and for the classification loss, M=3, the corresponding categories are sample addition window, detection window and label line window; This represents the true class of the sample; when sample i belongs to class c... =1, otherwise 0; This represents the probability that sample i belongs to class c, as predicted by the network. The information processing module is used to train the image classification model using the training sample set and the multi-task loss function. The image classification model uses the maximum probability of all possible detection boxes in the image to represent the probability of the presence of the sample addition window, the detection window, the C-line, and the T-line in the image. For positive and negative images of the type A coronavirus antigen detection, the sample addition window, the detection window, and the marker line window all exist. For irrelevant images, none of these three windows exist. The module determines the model parameters of the image classification model to classify the image to be classified using the image classification model.

9. The apparatus according to claim 8, characterized in that, The information processing module is also used to determine the disease type corresponding to the image classification model based on the usage environment of the image classification model; Based on the disease type, determine the bounding box type corresponding to the image classification model, and the classification loss function corresponding to the bounding box type; Obtain the confidence loss function of the bounding box; The multi-task loss function of the image classification model is calculated based on the classification loss function and the confidence loss function.

10. The apparatus according to claim 8, characterized in that, The information processing module is further configured to perform local augmentation processing on the second original sample image according to the usage environment of the image classification model to obtain a local augmented image, so as to realize spatial filtering on the local neighborhood information of the second original sample image; The second original sample image is subjected to global augmentation processing to obtain a global augmented image, thereby adjusting the sharpness of the second original sample image.

11. The apparatus according to claim 8, characterized in that, The information processing module is also used to determine a dynamic noise threshold that matches the usage environment of the image classification model; The training sample set is denoised according to the dynamic noise threshold to form a training sample set that matches the dynamic noise threshold; or, A fixed noise threshold corresponding to the image classification model is determined, and the training sample set is denoised according to the fixed noise threshold to form a training sample set that matches the fixed noise threshold.

12. The apparatus according to claim 8, characterized in that, The information processing module is also used to adjust the network parameters of the image classification model based on the feature vector of the training sample set and the multi-task loss function; The network parameters of the image classification model are determined when the loss functions of different dimensions corresponding to the image classification model reach the corresponding convergence conditions, so as to make the parameters of the image classification model compatible with the usage environment.

13. An image classification device, characterized in that, The device includes: The data transmission module is used to acquire the image to be classified and disease type information, wherein the disease type information includes: Disease type identifier, and the corresponding confirmed image bounding box status; The data processing module is used to extract features from the training sample set through the feature extraction network in the image classification model to obtain the image features of the training sample set. The data processing module is used to classify the image features of the training sample set through the image classification network in the image classification model to obtain the classification result of the image to be classified. The image classification model is trained by the method described in any one of claims 1 to 5; The data processing module is used to issue a confirmed alarm message when the classification result of the image to be classified is the same as the bounding box state of the confirmed image.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, configured to implement the image classification model training method according to any one of claims 1 to 5, or the image classification method according to any one of claims 6 to 7, when executing executable instructions stored in the memory.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the image classification model training method according to any one of claims 1 to 5, or the image classification method according to any one of claims 6 to 7.

16. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the image classification model training method according to any one of claims 1 to 5, or the image classification method according to any one of claims 6 to 7.

Images