Image classification model training method, image classification method, device and apparatus

Abstract

The application provides an image classification model training method, comprising the following steps: determining network parameters of a first image classification network in an image classification model, iteratively updating network parameters of a second image classification network through a training sample set to obtain intermediate parameters of the image classification model; processing a second original sample image through a supervision model to obtain model adjustment parameters, adjusting the intermediate parameters of the image classification model to obtain final model parameters of the image classification model, thereby stably improving the accuracy of image classification model training under the premise of reducing the total amount of training data and without labeling, reducing the overfitting of the image classification model and enhancing the generalization ability of the image classification model. The application also provides an image classification model training device, an electronic device, a computer program product and a storage medium. The embodiments of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an image classification model training method, an image classification method, an image classification device, an electronic device, a computer program product, and a storage medium. BACKGROUND

[0002] Various types of recognition based on deep learning have always been an important tool for solving large amounts of data division in various application scenarios. For example, in image, natural language processing and other application scenarios, large-scale classification and recognition are realized on a large amount of data, so as to quickly and accurately obtain relevant classification prediction results and accelerate the function implementation of the application scenario.

[0003] In the field of computer vision (CV), various image processing techniques based on artificial intelligence have been significantly developed and have a wide range of applications. Computer vision can be applied to a variety of different image processing tasks, such as image classification and panoramic classification of images, etc. For these image processing tasks, it is usually necessary to obtain a feature representation of the image for further processing. Therefore, the accuracy of the obtained feature representation in describing the image affects the accuracy of the image processing task. In addition, it is also desirable for the model used to generate the feature representation of the image to be scalable and applicable to a variety of different image processing tasks. SUMMARY

[0004] Therefore, the embodiments of the present application provide an image classification model training method, an image classification method, an image classification device, an electronic device, a computer program product, and a storage medium, which can realize stable improvement of the accuracy of image classification model training under the premise of reducing the total amount of training data and without the need for re-labeling data, alleviate overfitting of the image classification model, save the training time of the image classification model, facilitate deployment of the trained image classification model in a terminal, and realize large-scale application of the image classification model.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide an image classification model training method, comprising:

[0007] obtaining a first original sample image without a label and a second original sample image carrying a label;

[0008] performing image processing on the first original sample image to form an initial training sample set of the image classification model;

[0009] performing data encoding processing on the initial training sample set to obtain a training sample set;

[0010] The information processing module is configured to process the first original sample image through the image classification model to obtain a first image classification network parameter of the image classification model.

[0011] The information processing module is configured to update the network parameter of the second image classification network through the training sample set according to the network parameter of the first image classification network to obtain an intermediate parameter of the image classification model.

[0012] The information processing module is configured to process the second original sample image through the supervised model to obtain a model adjustment parameter.

[0013] The information processing module is configured to adjust the intermediate parameter of the image classification model according to the model adjustment parameter to obtain a final model parameter of the image classification model.

[0014] The embodiment of the present application further provides an image classification method, and the method comprises the following steps:

[0015] acquiring an image to be processed, wherein the image to be processed comprises at least one of the following:

[0016] a medical image, a road information monitoring image and an invoice image;

[0017] processing the image to be processed through an image classification model to obtain an image classification result.

[0018] The embodiment of the present application further provides an image classification model training device, and the device comprises the following:

[0019] an information transmission module configured to acquire a first original sample image without a label and a second original sample image carrying a label;

[0020] an information processing module configured to process the first original sample image through image processing to form an initial training sample set of the image classification model;

[0021] The information processing module is configured to process the initial training sample set through data coding to obtain a training sample set.

[0022] The information processing module is configured to process the training sample set through the image classification model to determine a network parameter of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network.

[0023] The information processing module is configured to update the network parameter of the second image classification network through the training sample set according to the network parameter of the first image classification network to obtain an intermediate parameter of the image classification model.

[0024] The information processing module is configured to process the second original sample image by using a supervised model to obtain model adjustment parameters.

[0025] The information processing module is configured to adjust intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

[0026] In the above scheme, the information processing module is configured to perform local augmentation processing on the first original sample image according to an application environment of the image classification model to obtain a local augmented image, so as to perform spatial filtering on local neighborhood information of the first original sample image.

[0027] The information processing module is configured to perform global augmentation processing on the first original sample image to obtain a global augmented image, so as to adjust a color of the first original sample image.

[0028] In the above scheme, the information processing module is configured to determine a dynamic noise threshold value matched with a use environment of the image classification model.

[0029] The information processing module is configured to perform denoising processing on the initial training sample set according to the dynamic noise threshold value to form an initial training sample set matched with the dynamic noise threshold value; or

[0030] The information processing module is configured to determine a fixed noise threshold value corresponding to the image classification model, and perform denoising processing on the initial training sample set according to the fixed noise threshold value to form an initial training sample set matched with the fixed noise threshold value.

[0031] In the above scheme, the information processing module is configured to perform structural division processing on different training samples in the initial training sample set according to an application environment of the image classification model to obtain at least two data blocks.

[0032] The information processing module is configured to perform extension processing on the at least two data blocks to obtain a training sample set, so as to adjust sample positions of the initial training sample set.

[0033] In the above scheme, the information processing module is configured to determine a first loss function matched with the first image classification network.

[0034] The information processing module is configured to bring a training sample in the training sample set into the first loss function.

[0035] The information processing module is configured to determine network parameters of the first image classification network when the first loss function satisfies a first convergence condition.

[0036] In the above scheme, the information processing module is configured to determine a supervised model corresponding to an application environment of the image classification model.

[0037] The application environment of the image classification model includes at least one of the following:

[0038] Invoice classification, medical image classification, and road information monitoring image classification.

[0039] The second original sample image is classified by the supervised model, and compared with the label of the second original sample image to obtain model adjustment parameters.

[0040] The embodiment of the application further provides an image classification device, and the device comprises:

[0041] An image transmission module is configured to acquire an image to be processed, wherein the image to be processed includes at least one of the following:

[0042] Medical images, road information monitoring images, and invoice images.

[0043] An image processing module is configured to classify the image to be processed by an image classification model to obtain an image classification result.

[0044] The embodiment of the application further provides an electronic device, and the electronic device comprises:

[0045] A memory is configured to store executable instructions.

[0046] A processor is configured to run the executable instructions stored in the memory to implement the image classification model training method.

[0047] The embodiment of the application further provides a computer readable storage medium, which stores executable instructions, and the executable instructions are executed by a processor to implement the image classification model training method.

[0048] The embodiment of the application has the following beneficial effects:

[0049] The application obtains a first original sample image without a label and a second original sample image carrying a label; image processing is performed on the first original sample image to form an initial training sample set of an image classification model; data coding processing is performed on the initial training sample set to obtain a training sample set; the training sample set is processed by the image classification model to determine network parameters of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network; the network parameters of the second image classification network are iteratively updated by the training sample set according to the network parameters of the first image classification network to obtain intermediate parameters of the image classification model; the second original sample image is processed by a supervision model to obtain model adjustment parameters; the intermediate parameters of the image classification model are adjusted based on the model adjustment parameters to obtain final model parameters of the image classification model, thereby, the accuracy of image classification model training can be stably improved under the premise of reducing the total amount of training data and without re-performing data labeling, overfitting of the image classification model is reduced, the training time of the image classification model is saved, the image classification model trained in time is facilitated to be deployed in a terminal, and large-scale application of the image classification model is realized. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a use environment schematic diagram of the image classification model training method provided by the embodiment of the application;

[0051] Figure 2 is a component structure schematic diagram of the electronic device provided by the embodiment of the application;

[0052] Figure 3 is an optional flow schematic diagram of the image classification model training method provided by the embodiment of the application;

[0053] Figure 4 is a model structure schematic diagram of the image processing model in the embodiment of the application;

[0054] Figure 5 is an optional process schematic diagram of the image classification model training method in the embodiment of the application;

[0055] Figure 6 is a structure schematic diagram of the first image classification network and the second image classification network in the embodiment of the application;

[0056] Figure 7 is an optional flow schematic diagram of the image classification model training method provided by the application;

[0057] Figure 8An optional flowchart of the image classification model training method provided by the embodiment of the present application is shown in the figure.

[0058] Figure 9 An effect diagram of image classification in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0060] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0061] Before the embodiment of the present application is further described in detail, the terms and terms related to the embodiment of the present application are described, and the terms and terms related to the embodiment of the present application are applicable to the following explanation.

[0062] 1) based on, used to represent the condition or state on which the operation is performed, when the dependent condition or state is met, one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of the multiple operations performed.

[0063] 2) client, a carrier for realizing specific functions in a terminal, for example, a mobile client (APP) is a carrier for realizing specific functions in a mobile terminal, for example, a function of performing online live broadcast (video push stream) or a function of playing online video.

[0064] 3) Convolutional Neural Networks (CNN) is a kind of feed forward neural network (Feed forward Neural Networks) containing convolution calculation and having a deep structure, which is one of the representative algorithms of deep learning (deep learning). Convolutional Neural Networks has representation learning ability and can perform shift-invariant classification on input information according to its hierarchical structure.

[0065] 4) Model training, multi-classification learning is performed on the image data set. The model can be constructed using deep learning frameworks such as Tensor Flow, torch, etc., and multi-layer combinations of neural network layers such as CNN are used to form a multi-image classification model. The input of the model is a three-channel or original channel matrix formed by reading the image using tools such as openCV, and the output of the model is a multi-classification probability. The final image classification result is output through algorithms such as softmax. During training, the model approaches the correct trend through objective functions such as cross-entropy.

[0066] 5) Neural Network (NN): Artificial Neural Network (ANN), simply neural network or neural network, in the field of machine learning and cognitive science, is a mathematical model or computational model that simulates the structure and function of biological neural networks (animal central nervous system, especially brain), used to estimate or approximate functions.

[0067] 6) Contrastive loss: contrastive loss function, which can learn a mapping relationship that can make the same class but distant points in high-dimensional space become close after mapping to low-dimensional space, and different classes but close points become further apart in low-dimensional space. The result is that in low-dimensional space, points of the same class will produce clustering effect, and different classes will be separated. Similar to fisher dimensionality reduction, but fisher dimensionality reduction does not have the effect of out-of-sample extension, and cannot act on new samples.

[0068] 7) Soft max: normalized exponential function, a generalization of the logistic function. It can "compress" a K-dimensional vector containing any real number into another K-dimensional real vector, so that each element is in the range [0, 1], and the sum of all elements is 1.

[0069] The image classification model training method provided by the embodiment of the application is described below. Figure 1 The application environment of the image classification model training method in the embodiment of the application is shown in Figure 1The terminal (including the terminal 10-1 and the terminal 10-2) is provided with a client of an application software with a face living body detection function, and the server is deployed with a trained image classification model to realize the verification of the face on the terminal side. For example, in the field of financial payment, the user can transfer money, pay or modify the account information through the smart phone, and the operation of identity verification can be realized by detecting the face living body of the user. In this process, the terminal device uploads the face image or video to be detected to the server, or the server directly calls the face image or face video to be detected in the database, and then uses the trained image classification model to verify the received face image or face video to obtain the classification result (real face image and attack image) for detection. The server can feed back the detection result to the terminal device, or keep the detection result locally for other business applications or processing. The terminal connects the server 200 through the network 300, which can be a wide area network or a local area network, or a combination of the two, and uses a wireless link to realize data transmission.

[0070] As an example, the server 200 is used to deploy an image classification model training device to realize the image classification model training method provided by the application. Of course, before the image classification model is used to process different face images to generate corresponding classification results, the image classification model needs to be trained. Specifically, the method comprises the following steps: obtaining a first original sample image without a label and a second original sample image carrying a label; performing image processing on the first original sample image to form an initial training sample set of the image classification model; performing data coding processing on the initial training sample set to obtain a training sample set; processing the training sample set through the image classification model to determine the network parameters of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network; according to the network parameters of the first image classification network, iteratively updating the network parameters of the second image classification network through the training sample set to obtain intermediate parameters of the image classification model; processing the second original sample image through a supervision model to obtain model adjustment parameters; and adjusting the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

[0071] Of course, the image classification model training apparatus provided by the present application can be applied to a virtual resource or a physical resource to perform a financial activity or an information interaction environment of a social software, and a payment environment of a physical financial resource (including but not limited to a face detection environment in various types of payment of a physical financial resource), in which financial information of different data sources is usually processed in the financial activity of various types of physical financial resources or the payment of a virtual resource, and finally a detection result matched with the target user is presented on a user interface (UI), to determine whether the detected image is a user's live face image or attack information. The face classification result (for example, judging that the current detection is attack information) obtained by the user in the current display interface can also be called by other application programs.

[0072] In some embodiments, the terminal can be a smart device such as a mobile phone, or a vehicle-mounted intelligent system. The image classification model training method provided by the present application can be used as a cloud service to serve customers of various types (encapsulated in a vehicle-mounted terminal or encapsulated in different mobile electronic devices), and the specific use scenarios are not limited in the present application. Among them, as a cloud service provided to enterprise customers, it helps to train image classification models to detect the types and positions, speeds of vehicles, positions and speeds of pedestrians, and other information in the road conditions.

[0073] Among them, the image classification model training method provided by the embodiments of the present application is realized based on artificial intelligence. Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0074] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0075] In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned speech processing technologies and machine learning and the like. For example, the speech recognition technology (Automatic Speech Recognition, ASR) in the speech technology can be involved, which includes speech signal preprocessing, speech signal frequency analyzing, speech signal feature extraction, speech signal feature matching / recognition, speech training and the like.

[0076] For example, machine learning (Machine learning, ML) can be involved, which is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. It is specially studied how the computer simulates or realizes the learning behavior of human beings to obtain new knowledge or skills, reorganizes the existing knowledge structure to constantly improve the performance. Machine learning is the core of artificial intelligence and is the fundamental approach to making computers intelligent, and its application is widespread in various fields of artificial intelligence. Machine learning usually includes technologies such as deep learning, and deep learning includes artificial neural networks such as convolutional neural network (Convolutional Neural Network, CNN), recurrent neural network (Recurrent Neural Network, RNN), deep neural network (Deep neural network, DNN) and the like.

[0077] Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool, and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item may have its own identification mark in the future, and it needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system support, which can only be realized through cloud computing.

[0078] The so-called artificial intelligence cloud service, also commonly known as AI as a service (AIaaS), is a mainstream service mode of artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service mode is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through API interface, and some experienced developers can also use the AI framework and AI infrastructure provided by the platform to deploy and maintain their own cloud artificial intelligence services.

[0079] The scheme provided by the embodiments of the present application relates to natural language processing, machine learning, artificial intelligence cloud service and other technologies of artificial intelligence, which is specifically explained by the following embodiments.

[0080] The image classification model training method and the image classification method provided by the embodiments of the present application will be explained by combining the exemplary application and implementation of the terminal provided by the embodiments of the present application.

[0081] The structure of the electronic device of the embodiments of the present application will be described in detail below. The electronic device can be implemented in various forms, such as a special terminal with image classification processing function, an electronic device with image classification function or a cloud server, for example, the server 200 in the foregoing Figure 1 . Figure 2 The schematic diagram of the composition structure of the electronic device provided by the embodiments of the present application can be understood that Figure 2 only the exemplary structure of the electronic device is shown, not all structures, and part or all of the structures shown can be implemented according to needs. Figure 2

[0082] ​The electronic device provided in the embodiments of the present application includes at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the electronic device 20 are coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection communication between the components. In addition to the data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 205 in the Figure 2

[0083] The user interface 203 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad or a touch screen, etc. The terminal in the embodiments of the present application includes but is not limited to a mobile phone, a computer, a smart voice interactive device, a smart household appliance, a vehicle-mounted terminal, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, etc. When the image classification model training method provided by the present application is executed through different terminals, the specific use scenarios are not limited by the present application

[0084] It can be understood that the memory 202 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The memory 202 in the embodiments of the present application can store data to support the operation of the terminal (such as 10-1). Examples of the data include any computer programs for operating on the terminal (such as 10-1), such as an operating system and an application program. The operating system contains various system programs, such as a framework layer, a core library layer, a driver layer, etc., for realizing various basic services and processing hardware-based tasks. The application program can include various application programs.

[0085] In some embodiments, the image classification model training apparatus provided by the embodiments of the present application can be realized in a combination of software and hardware. As an example, the image classification model training apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image classification model training method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.

[0086] ​As an example of the image classification model training device provided in this embodiment of the invention, which adopts a combination of hardware and software, the image classification model training device provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus 205) to complete the image classification model training method provided in this embodiment of the invention.

[0087] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0088] As an example of the hardware implementation of the image classification model training device provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the image classification model training method provided in this embodiment of the invention.

[0089] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the electronic device 20. Examples of such data include: any executable instructions for operation on the electronic device 20, such as executable instructions that implement the image classification model training method of this embodiment of the invention can be included in the executable instructions.

[0090] In other embodiments, the image classification model training device provided in this invention can be implemented in software. Figure 2An image classification model training apparatus 2020 stored in the memory 202 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules. As an example of the program stored in the memory 202, the image classification model training apparatus 2020 can include the following software modules included in the image classification model training apparatus 2020:

[0091] An information transmission module 2081 is configured to obtain a first original sample image without a label and a second original sample image carrying a label.

[0092] An information processing module 2082 is configured to perform image processing on the first original sample image to form an initial training sample set of the image classification model.

[0093] The information processing module 2082 is configured to perform data encoding processing on the initial training sample set to obtain a training sample set.

[0094] The information processing module 2082 is configured to process the training sample set by the image classification model to determine network parameters of a first image classification network in the image classification model, wherein the image classification model includes the first image classification network and a second image classification network.

[0095] The information processing module 2082 is configured to iteratively update network parameters of the second image classification network by the training sample set according to the network parameters of the first image classification network to obtain intermediate parameters of the image classification model.

[0096] The information processing module 2082 is configured to process the second original sample image by a supervised model to obtain model adjustment parameters.

[0097] The information processing module 2082 is configured to adjust the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

[0098] After the training is completed, the image classification model can continue to be deployed in the electronic device 20, and therefore, the electronic device 201 can further include:

[0099] An image transmission module 2083 is configured to obtain an image to be processed, wherein the image to be processed includes at least one of:

[0100] a medical image, a road information monitoring image, and an invoice image;

[0101] An image processing module 2084 is configured to classify the image to be processed by the image classification model to obtain an image classification result.

[0102] According toFigure 2 In an aspect of the present application, the electronic device shown, the present application also provides a computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementation manners of the image classification model training method provided by the present application.

[0103] In combination with Figure 2 The image classification model training device shown illustrates the image classification model training method provided by the embodiment of the present application, and the image classification model training device is used to execute the image classification model training method. The image classification model training device comprises a processor and a memory. The memory is configured to store a computer program. The processor is configured to execute the computer program stored in the memory. The computer program comprises the steps shown in the following figure. Figure 3 , Figure 3 An optional flowchart of the image classification model training method provided by the embodiment of the present application can be understood as Figure 3 The steps shown can be executed by various electronic devices running the image classification model training device, for example, can be a special terminal, a server or a server cluster with image classification processing function. The steps shown in the following figure will be described. Figure 3

[0104] Step 301: The image classification model training device acquires the first original sample image without label and the second original sample image carrying label.

[0105] ​In some embodiments of the present application, the second original sample image refers to an image that can provide rich annotation information, and the first original sample image refers to an image that lacks annotation information in the field where the test data set is located or an unannotated image collected by a collection machine in real time. In the training process of the image classification model, the second original sample image and the first original sample image describe the same scene and solve the same task, and the data distribution of the second original sample image and the first original sample image is related but different, so the second original sample image and the first original sample image can be very different in appearance style, but have many similarities in classification results, such as spatial layout and local context. Through the field adaptation training of the image classification model, the image classification model that performs well in the second original sample image can be applied to the first original sample image. The second original sample image can be collected by an image collection device, annotated by a professional, and then sent to the image classification model training device. For example, the second original sample image can specifically be a medical image, so the second original sample image can be collected by various medical image collection devices such as a computed tomography (CT) or a nuclear magnetic resonance imaging device, and annotated by an image physician to provide to the image classification model training device, that is, the medical image sample sent by the medical image collection device can be received.

[0106] In some embodiments of the present application, the second original sample image can be a street view image, so the second original sample image can be collected by a monitoring device on the street and annotated by a professional to provide to the image classification model training device, and a street view image synthesized by a game engine from a game video can also be collected as the second original sample image. The medical image refers to an image of internal tissue of a living body or a part of a living body obtained in a non-invasive manner in medical or medical research, such as images of the brain, intestines and stomach, liver, heart, throat, and vagina of a human body. These images can be CT images, nuclear magnetic resonance images, or positron emission tomography images, etc. The living body refers to an independent individual with a life form, such as a human or an animal, etc. The second original sample image can refer to an image that has been collected by a medical image collection device and obtained through various means, such as obtaining the second original sample image from a database or a network, etc. The second original sample image can be an image sample annotated by a professional for specific meaning, or an image sample without any processing.

[0107] In some embodiments of the present application, taking the five-wu facial image living body detection of the image classification model as an example, after obtaining the user image collected by the terminal, the face detection technology can be used to frame the region where the user's face is located, and the region is enlarged by 1.8 times to obtain more background content, and the face image including the background content is cropped; for example, the following method can be used: using a face detection algorithm, the face position of the target object is framed; using a five-wu positioning algorithm, the feature points of the eyes, mouth, nose, and other facial features are marked; and the face image including the background content is cut according to the detected face position. Then, the depth map corresponding to the face is calculated by the deep estimation network.

[0108] Step 302: The image classification model training device performs image processing on the first original sample image to form an initial training sample set of the image classification model.

[0109] In some embodiments of the present application, the image processing on the first original sample image to form an initial training sample set of the image classification model can be realized by the following method:

[0110] According to the application environment of the image classification model, the first original sample image is locally augmented to obtain a locally augmented image to realize the spatial filtering of the local neighborhood information of the first original sample image; and the first original sample image is globally augmented to obtain a globally augmented image to realize the color adjustment of the first original sample image. Wherein, the first original sample image is denoted as X, and two types of transformations are performed, which are: global augmentation processing Global and local augmentation processing Local. The global augmentation processing can derive a transformation function that maps the input color to the output color. The local augmentation processing can perform spatial filtering according to the local neighborhood information to determine the pixel color. In some embodiments of the present application, in order to enhance the sample processing effect, when performing image processing, the enhancement of the local augmentation processing X1 direction is only Local transformation, and the Global and Local transformations are performed on the globally augmented processing result X2 side, so that the image classification model can better learn the relationship between the global augmentation processing Global and the local augmentation processing Local. It should be noted that the examples of data augmentation operations include but are not limited to size scaling, color jittering, Gaussian filtering, etc., and the present application does not make specific limitations thereto.

[0111] In some embodiments of the present application, a dynamic noise threshold value matched with the use environment of the image classification model is determined.

[0112] According to the dynamic noise threshold, the initial training sample set is denoised to form an initial training sample set matched with the dynamic noise threshold; wherein the dynamic noise threshold matched with the use environment of the image classification model is not the same due to the difference in the use environment of the image classification model, for example, the dynamic noise threshold matched with the use environment of the image classification model in the medical image classification needs to be smaller than the dynamic noise threshold in the invoice classification environment.

[0113] In some embodiments of the present application, a fixed noise threshold corresponding to the image classification model can also be determined, and the initial training sample set is denoised according to the fixed noise threshold to form an initial training sample set matched with the fixed noise threshold. Wherein when the image classification model is fixed in the corresponding hardware mechanism, for example, the vehicle terminal, the use environment is to classify the objects in the road image, due to the single noise, by fixing the fixed noise threshold corresponding to the image classification model, the training speed of the image classification model can be effectively improved, and the waiting time of the user is reduced.

[0114] Step 303: The image classification model training device performs data encoding processing on the initial training sample set to obtain a training sample set.

[0115] In some embodiments of the present application, the data encoding processing on the initial training sample set to obtain a training sample set can be realized by the following way:

[0116] According to the application environment of the image classification model, different training samples in the initial training sample set are structurally divided to obtain at least two data blocks; the at least two data blocks are extended to obtain a training sample set, so as to adjust the sample position of the initial training sample set. Wherein, according to the previous embodiment, for X1 and X2, they can be divided into multiple patches according to the structure of N*N, wherein the preferred value of N is 16 or 14. And the block structure is straightened and extended, tiled and unfolded. By adjusting the sample position of the initial training sample set, the accuracy of the image classification model can be improved by 3-7%, so that the user can obtain better use experience.

[0117] Step 304: The image classification model training device determines the network parameters of the first image classification network in the image classification model by processing the training sample set through the image classification model, wherein the image classification model comprises a first image classification network and a second image classification network.

[0118] Reference Figure 4 , Figure 4FIG. 1 is a schematic diagram of a model structure of an image processing model in an embodiment of the present application. The image classification model includes a first image classification network and a second image classification network. The model structures of the first image classification network and the second image classification network are the same, and are both ViT (Vision Transformer) model structures. Specifically, in order to enable the Transformer model to receive an image, the image to be input can be processed by a preset grid to divide it into a plurality of sub-images, such as the 16 sub-images Seg1-Seg16 shown in FIG. 1, which can be divided by a 4*4 preset grid. Then, the image feature extraction network can be implemented based on the convolutional neural network model, so as to convert each of the sub-images into a vector by the convolutional neural network. For example, the image feature extraction network in this step can be a two-layer convolutional neural network to perform image encoding on different sample images. Figure 4

[0119] In some embodiments of the present application, Figure 4 The model structure shown in FIG. 1 includes n layers stacked together: layer 1, …, layer n. The image is divided into N image blocks, each of which corresponds to an input position of the network. The N image blocks are respectively linearly mapped or flattened to generate a label [N, C]. The label [N, C] is input to the n layers stacked together for processing. Each layer can include two normalization modules, an MHA (multi head self-attention) block, and an MLP (Multi Perceptron) module. The normalization module can keep the input distribution the same.

[0120] The MHA module can be used to extract semantic information. The MLP module can be a two-layer fully connected layer for information induction and integration. The label [N, C] is input to the first normalization module for normalization processing. The processing results thereof are input to the MHA module through three heads of the MHA module for semantic information extraction. The extraction results thereof are matrix added with the label [N, C], and the addition results thereof are input to the second normalization module for normalization processing. The processing results thereof are input to the MLP module for information induction and integration. The integration results thereof are matrix added with the first matrix addition results, and the output label [N, C] has more semantic information than the input label [N, C].

[0121] ​In some embodiments of the present application, when the ViT model is used to classify the first sample image, the first sample image is first segmented into image blocks of a fixed size, and then each image block is transformed into an initial image feature through linear transformation, that is, each image block is encoded into a processing unit (token), and each processing unit has sequential information.

[0122] In combination with the foregoing embodiments, when the dimension of X is (B, C, H, W), where B is a batch size, C is three channels, H and W are the height and width of the image respectively, and the output is (B, N, E), B is still a batch size, N is the number of paths after each image is cut into different paths, and E is an embedding size, each path is converted into a vector through a fully connected network, and E is the length of the vector, which can also be understood as the number of features of each path according to the principle of convolution

[0123] Step 305: The image classification model training device iteratively updates the network parameters of the second image classification network through the training sample set according to the network parameters of the first image classification network, and obtains intermediate parameters of the image classification model.

[0124] Reference Figure 5 , Figure 5 An optional process diagram for the image classification model training method in the embodiments of the present application is shown in the figure, which specifically includes the following steps:

[0125] Step 501: Determine a first loss function matched with the first image classification network.

[0126] The first image classification network can be a student network, that is, Student gθs, and the second image classification network can be a teacher network, that is, Teacher gθs.

[0127] The first image classification network can be trained by using a cross-entropy loss function according to formula 1:

[0128]

[0129] Where N S is the number of training sample set images, H and W are the height and width of the image respectively, and C is the number of categories. is a one-hot vector, where different categories of data in the source domain are encoded by one-hot encoding (One-Hot Encoding), specifically including: using an N-bit state register to encode N states, each state has its own register bit, and at any time, only one bit is valid.

[0130] Step 502: Substitute the training samples from the training sample set into the first loss function.

[0131] Step 503: When the first loss function satisfies the first convergence condition, determine the network parameters of the first image classification network.

[0132] Step 504: In response to the network parameters of the first image classification network, determine a second loss function that matches the second image classification network.

[0133] Step 505: Substitute the training samples from the training sample set into the second loss function.

[0134] Step 506: When the second loss function satisfies the second convergence condition, determine the network parameters of the second image classification network.

[0135] Continue to refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of the first image classification network and the second image classification network in an embodiment of the present invention. Since training the classification model requires a large number of positive and negative samples, positive samples are relatively easy to obtain, as many labeled images can be directly used as positive samples. However, labeling negative samples is costly. The present invention can obtain negative samples through a manually constructed method. Specifically:

[0136] pass Figure 6 The image augmentation layer shown can randomly generate corresponding negative samples from images aligned in a shuffled sample set. Taking a practical VITs structure as the first and second image classification networks as an example, Student g... θs With Teacher g θs After obtaining the training samples, the first image classification network processes them and outputs a feature map.

[0137] The image classification network then uses the sofmax function to generate a K-dimensional prediction vector. The sofmax function here is referenced in Formula 2:

[0138]

[0139] Thus, the output result pred1 of the first image classification network is obtained, where P s (x) (i) Let gθ be the K-dimensional prediction vector obtained from the prediction. s (x) (i) For model output, τ s For parameter adjustment. Reference. Figure 6As shown in the model structure, the first image classification network directly enters the sofmax function after outputting the result, and the second image classification network will enter the τ s of the Teacher model is less than that of the Student model s . As a result, the probability distribution of the Teacher model can be made more sharp, and the centralized processing can compensate for the results brought by the sharp behavior. The calculation process of the centralized processing is referred to formula 3, where gθ(xi) represents the input of the logistic regression function layer (Softmax layer) for the sample X, such as the feature representation or the feature representation h′B represents the number of images in a batch of training samples. m is a preset proportion parameter, and the value of m can be between 0 and 1, for example, 0.5. c represents the center point. The center point c is updated each time using the above formula (5).

[0140]

[0141] After updating the value of C, the predicted value is calculated according to formula 4:

[0142]

[0143] During training, the parameters of the model of the second image classification network are updated iteratively, and reference is made to formula 5:

[0144] gt.params = m * gt.params + (1-m) * gt.params. Formula 5

[0145] Wherein, m is an empirical value, preferably 0.99, params is the parameter of the second image classification network after each iteration, and gt.params is the final parameter of the second image classification network.

[0146] Thus, the output value obtained by the first image classification network is directly subjected to the confidence of the sofmax function: P1; the output value obtained by the second image classification network is subjected to the centralized calculation, which can effectively prevent the model from collapsing and converging to a local extremum, and then subjected to the sofmax function to obtain the confidence P2. Combined Figure 6 As shown, the weights of the Student g θs and the Teacher g θs are constantly updated, so that the output of the Student g θs is constantly the Teacher g θs , and finally the training of the image classification model is completed.

[0147]

[0148]

[0149] Step 306: The image classification model training device processes the second original sample image through a supervised model to obtain model adjustment parameters.

[0150] Step 307: The image classification model training device adjusts intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

[0151] In reference to Table 1, the image classification model can achieve an identification accuracy of more than 93% within 5 epochs.

[0152] Table 1

[0153]

[0154] When the image classification model training is completed, the image classification model can be deployed in a server and used to process images to be processed. For reference Figure 7 , Figure 7 An optional flowchart of the image classification model training method provided in the present application is shown in FIG. 7. Figure 5 After the model training is completed, the model can be deployed in a corresponding server, and specifically includes the following steps:

[0155] Step 507: Determine a supervised model corresponding to an application environment of the image classification model.

[0156] The application environment of the image classification model includes at least one of the following:

[0157] Invoice classification, medical image classification, and road information monitoring image classification.

[0158] Step 508: Process the image to be processed through the second image classification model to obtain an image segmentation result.

[0159] To better illustrate the processing process of the image classification model training method provided in the present application, the use environment of the image classification model training method provided in the present application is described below by taking the use of the image classification model to process invoices as an example. For reference Figure 1 to the use scenario diagram, the image classification model training method provided in the present application can be used as a cloud service to serve customers of the type (encapsulated in an invoice identification applet or encapsulated in different applet cloud servers). The specific use scenarios are not limited in the present application. Specifically, the image classification model training method provided in the present application can be provided as a cloud service to enterprise customers to help the enterprise customers detect various invoices.

[0160] In some embodiments of the present application, for reference Figure 8 , Figure 8An optional flowchart of the image classification model training method provided by the embodiment of the present application specifically comprises the following steps:

[0161] Step 801: Obtain a first original invoice image without a label and a second original invoice image carrying a label.

[0162] Step 802: Perform image processing on the first original invoice image to form an initial training sample set of the image classification model.

[0163] Step 803: Perform data coding processing on the initial training sample set to obtain a training sample set.

[0164] Step 804: Process the training sample set by the image classification model to determine the network parameters of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network.

[0165] Step 805: According to the network parameters of the first image classification network, iteratively update the network parameters of the second image classification network by the training sample set to obtain intermediate parameters of the image classification model.

[0166] Step 806: Process the second original invoice image by a supervision model to obtain model adjustment parameters.

[0167] Step 807: Adjust the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

[0168] Step 808: Perform classification processing on the uploaded invoice image.

[0169] Reference Figure 9 , Figure 9 The figure is an effect diagram of image classification in the embodiment of the present application, wherein through classification of the invoice image, 1) the name of the purchasing unit, the taxpayer registration number, the bank of opening an account and the account number can be determined; 2) the name of goods or services, the unit of measurement, the quantity, the unit price and the amount; 3) the name of the selling unit, the taxpayer registration number, the bank of opening an account and the account number; and 4) the financial special seal or the invoice special seal of the invoicing unit.

[0170] Beneficial technical effects:

[0171] The application obtains a first original invoice image without a label and a second original invoice image carrying a label; performs image processing on the first original invoice image to form an initial training sample set of an image classification model; performs data coding processing on the initial training sample set to obtain a training sample set; processes the training sample set by the image classification model to determine network parameters of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network; iteratively updates network parameters of the second image classification network by the training sample set according to the network parameters of the first image classification network to obtain intermediate parameters of the image classification model; processes the second original invoice image by a supervision model to obtain model adjustment parameters; adjusts the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model, thereby, the accuracy of the image classification model training can be stably improved under the premise of reducing the total amount of training data and without re-performing data labeling, overfitting of the image classification model is reduced, training time of the image classification model is saved, the image classification model trained in time is facilitated to be deployed in a terminal, and large-scale application of the image classification model is realized.

[0172] The above merely describes the embodiments of the present application and is not used to limit the protection scope of the present application, and any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an image classification model, characterized in that, The method comprises: acquiring a first original sample image without a label and a second original sample image carrying a label; performing image processing on the first original sample image to form an initial training sample set of an image classification model; performing data coding processing on the initial training sample set to obtain a training sample set; processing the training sample set through the image classification model to determine network parameters of a first image classification network in the image classification model, wherein the image classification model comprises the first image classification network and a second image classification network; iteratively updating network parameters of the second image classification network through the training sample set according to the network parameters of the first image classification network to obtain intermediate parameters of the image classification model, wherein the second image classification network is consistent with the first image classification network in terms of parameter structure, and parameter changes of the network parameters of the second image classification network are updated along with parameter changes of the network parameters of the first image classification network; processing the second original sample image through a supervised model to obtain model adjustment parameters; adjusting the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model.

2. The method of claim 1, wherein, The image processing on the first original sample image to form the initial training sample set of the image classification model comprises: performing local augmentation processing on the first original sample image according to an application environment of the image classification model to obtain a locally augmented image, so as to perform spatial filtering on local neighborhood information of the first original sample image; performing global augmentation processing on the first original sample image to obtain a globally augmented image, so as to adjust the color of the first original sample image.

3. The method of claim 1, wherein, The method further comprises: determining a dynamic noise threshold value matched with a use environment of the image classification model; performing denoising processing on the initial training sample set according to the dynamic noise threshold value to form an initial training sample set matched with the dynamic noise threshold value; or determining a fixed noise threshold value corresponding to the image classification model, and performing denoising processing on the initial training sample set according to the fixed noise threshold value to form an initial training sample set matched with the fixed noise threshold value.

4. The method of claim 1, wherein, The data coding processing on the initial training sample set to obtain the training sample set comprises: performing structural division processing on different training samples in the initial training sample set according to an application environment of the image classification model to obtain at least two data blocks; performing extension processing on the at least two data blocks to obtain the training sample set, so as to adjust the sample positions of the initial training sample set.

5. The method of claim 1, wherein, The processing of the training sample set through the image classification model to determine the network parameters of the first image classification network in the image classification model comprises: determining a first loss function matched with the first image classification network; inputting a training sample in the training sample set into the first loss function; When the first loss function satisfies a first convergence condition, network parameters of the first image classification network are determined.

6. The method of claim 1, wherein, The processing of the second original sample image by the supervised model includes: A supervised model corresponding to an application environment of the image classification model is determined. The application environment of the image classification model includes at least one of the following: Invoice classification, medical image classification, and road information monitoring image classification; The second original sample image is classified by the supervised model, and the model adjustment parameters are obtained by comparing the label of the second original sample image.

7. An image classification method characterized by, The method includes: An image to be processed is obtained, wherein the image to be processed includes at least one of the following: Medical images, road information monitoring images, and invoice images; An image classification model is used to classify the image to be processed to obtain an image classification result. The device includes:

8. An image classification model training apparatus, characterized by comprising: An information transmission module is configured to obtain a first original sample image without a label and a second original sample image carrying a label; An information processing module is configured to perform image processing on the first original sample image to form an initial training sample set of an image classification model; The information processing module is configured to perform data encoding processing on the initial training sample set to obtain a training sample set; The information processing module is configured to process the training sample set by using the image classification model to determine network parameters of a first image classification network in the image classification model, wherein the image classification model includes the first image classification network and a second image classification network; The information processing module is configured to iteratively update network parameters of the second image classification network by using the training sample set according to the network parameters of the first image classification network to obtain intermediate parameters of the image classification model, wherein the second image classification network has the same parameter structure as the first image classification network, and the parameter change of the network parameters of the second image classification network is updated according to the parameter change of the network parameters of the first image classification network; The information processing module is configured to process the second original sample image by using a supervised model to obtain model adjustment parameters; The information processing module is configured to adjust the intermediate parameters of the image classification model based on the model adjustment parameters to obtain final model parameters of the image classification model. The device includes:

9. An image classification apparatus characterized by comprising: An image transmission module is configured to obtain an image to be processed, wherein the image to be processed includes at least one of the following: Medical images, road information monitoring images, and invoice images; An image processing module is configured to classify the image to be processed by using an image classification model to obtain an image classification result. The electronic device includes: A memory is configured to store executable instructions; 10. An electronic device, comprising: ​ ​ A processor, when running executable instructions stored in the memory, implements the image classification model training method according to any one of claims 1 to 6, or implements the image classification method according to claim 7.

11. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the image classification model training method according to any one of claims 1 to 6, or implement the image classification method according to claim 7.

12. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: The executable instructions, when executed by a processor, implement the image classification model training method according to any one of claims 1 to 6, or implement the image classification method according to claim 7.

Images