Model training method and system, image recognition method and system, and medium

Abstract

The application provides a model training method and system, an image recognition method and system, and a medium. A sample image of computer vision is obtained. A sample energy map is calculated according to the sample image. The sample energy map includes energy data of the sample image. The sample image is input into a recognition model to obtain energy reconstruction data. The recognition model is trained according to an error between the energy reconstruction data and the energy data of the sample energy map. The application calculates a sample energy map of a sample image, quantifies image content importance using an energy map, changes the reconstruction target from'reconstructing pixels' to'reconstructing energy', and further improves the semantic level of the latent representation learned by an encoder and the feature robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to computer vision and deep learning technology, and especially relates to a model training method and system, an image recognition method and system, and a medium. BACKGROUND

[0002] In the field of artificial intelligence, the goal of deep learning of a machine is to enable the machine to have intelligent analysis and learning capabilities. For image analysis and learning models of a machine, the mainstream model training method currently uses a supervised learning method, the core idea of which is to use labeled data to provide strong supervision feedback information for the model. During training, the loss function, i.e., the strong supervision feedback information, is calculated, and the gradient backpropagation technique is used to continuously adjust the parameters of the model, so that the model gradually fits the target output distribution and converges.

[0003] However, supervised learning relies on expensive and rare human labels, and only a small part of the vast amount of data in specific scenarios in actual situations can be manually annotated, and a large amount of unlabeled data cannot be utilized by the supervised learning method. Moreover, the training of the model using a small amount of data may cause the model to have poor generalization and robustness and be vulnerable to noise attacks. SUMMARY

[0004] The model training method and system and the image recognition method and system provided by the present application improve the semantic level of the latent representation learned by the encoder by calculating the sample energy map of the sample image, quantifying the image content importance using the energy map, and changing the reconstruction target from "reconstructing pixels" to "reconstructing energy", thereby improving the feature robustness.

[0005] According to a first aspect of an embodiment of the present application, a model training method is provided, including the following steps:

[0006] Obtaining a sample image of computer vision;

[0007] Calculating a sample energy map according to the sample image; the sample energy map includes energy data of the sample image;

[0008] Inputting the sample image into a recognition model to obtain energy reconstruction data through prediction;

[0009] Training the recognition model according to the error between the energy reconstruction data and the energy data of the sample energy map.

[0010] In some embodiments of the present application, the sample energy map is calculated according to the sample image, including:

[0011] Traversing the sample content in the x-axis direction of the sample image and the x-axis direction of the sample image;

[0012] The gradient intensity of the sample image in the x-axis direction is calculated according to the energy calculation formula as the x-axis direction energy data of the sample energy map; the gradient intensity of the sample image in the y-axis direction is calculated according to the energy calculation formula as the y-axis direction energy data of the sample energy map; the data size of the energy of the sample image is in a positive correlation with the importance of the content of the sample.

[0013] In some embodiments of the present application, the sample image is input into a model for prediction to obtain energy reconstruction data, including:

[0014] The sample image is subjected to cutting, convolution and random masking operation to obtain local block encoding.

[0015] The local block encoding is input into an encoder adopting a Transformer structure to output encoded features of multiple feature dimensions.

[0016] The encoded features are input into an MLP regressor to obtain energy reconstruction data.

[0017] In some embodiments of the present application, the sample image is subjected to cutting, convolution and random masking operation to obtain local block encoding, including:

[0018] The sample image I is subjected to equal-scale blocking to obtain N number of local blocks; wherein the sample image I has a scale of H*W, each local block has a size of n*m, and the number of local blocks is

[0019] A convolution kernel is used The convolution operation is performed on all the local blocks to obtain N one-dimensional local block encodings: wherein c0 is the number of channels input into the encoder, and c is the number of local block encodings.

[0020] The random masking operation is performed on the local block encoding to obtain local block encoding including a mask block subset and a visible block subset; wherein the mask block subset MT={mt0, mt1, …, mt k}, and the visible block subset T={t0, t1, …, t N-k}.

[0021] In some embodiments of the present application, the local block encoding is input into an encoder adopting a Transformer structure to output encoded features of multiple feature dimensions, including:

[0022] According to the position information of the local block encoding in the image, position encoding PE={pe0, pe1, …, pe N} is calculated.

[0023] The position encoding and the local block encoding are added one by one to obtain local block encoding with position information

[0024] The local block with position information is input into an encoder with a Transformer structure, and encoded features with multiple representation dimensions are output wherein c z is a representation dimension.

[0025] In some embodiments of the present application, the encoded features are input into an MLP regressor to obtain energy reconstruction data, including:

[0026] The reconstruction target of the MLP regressor is to reconstruct the energy values of the mask region, and the encoded features are input into the MLP regressor to predict the normalized energy values of the mask blocks, and a mask block energy reconstruction result set MP={mp0,mp1,…,mp k} is obtained, wherein mp i ∈R n*m ;

[0027] Each mask block energy reconstruction result mp i is unfolded to obtain a one-dimensional vector mp′ i ∈R (n*m)*1 , and finally the energy reconstruction data MP′={mp′0,mp′1,…,mp′ k} is obtained.

[0028] In some embodiments of the present application, the model is trained according to the error between the energy reconstruction data and the energy data of the sample energy map, including:

[0029] The sample energy map is equally divided into blocks to obtain N local energy blocks energy i ∈R n*m ;

[0030] According to the position screening of the mask block subset MT, a mask block energy label set ME={me0,me1,…,me k} is determined; each mask block energy label me i is unfolded into a one-dimensional vector me′ i ∈R (n*m)*1 , and a new mask block energy label set ME′={me′0,me′1,…,me′ k} is obtained as the energy data of the final sample energy map;

[0031] According to the energy reconstruction data MP′={mp′0,mp′1,…,mp′ k} and the mask block energy label set ME′={me′0,me′1,…,me′ k}, the reconstruction error

[0032] Based on the reconstruction error l, the parameters of the encoder and MLP regressor are updated iteratively through gradient backpropagation.

[0033] According to a second aspect of the embodiments of this application, an image recognition method is provided, comprising:

[0034] Acquire the image to be detected;

[0035] A recognition model is used to predict objects in the image to be detected in order to obtain the annotation information of each target object in the image to be detected; wherein, the recognition model is obtained by any of the above model training methods.

[0036] According to a third aspect of the embodiments of this application, a model training system is provided, comprising:

[0037] Sample acquisition module: Used to acquire sample images for computer vision;

[0038] Energy Map Module: Used to calculate the sample energy map based on the sample image; the sample energy map includes the energy data of the sample image;

[0039] Energy Reconstruction Module: Used to input sample images into the recognition model for prediction to obtain energy reconstruction data;

[0040] Model training module: Used to train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map.

[0041] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement a model training method.

[0042] The model training method, system, and image recognition method, system, and medium of this application acquire sample images from computer vision; calculate a sample energy map based on the sample images; the sample energy map includes energy data of the sample images; input the sample images into a recognition model for prediction to obtain energy reconstruction data; and train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map. This application calculates the sample energy map of sample images, uses the energy map to quantify the importance of image content, and changes the reconstruction target from "reconstructing pixels" to "reconstructing energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness. Attached Figure Description

[0043] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0044] Figure 1 The diagram shows a schematic representation of the steps of a model training method according to an embodiment of this application.

[0045] Figure 2 The diagram illustrates the steps for obtaining energy reconfiguration data according to an embodiment of this application.

[0046] Figure 3 The diagram shows a schematic representation of the model training method according to an embodiment of this application.

[0047] Figure 4 The figure shows a schematic diagram of the steps of an image recognition method according to an embodiment of this application;

[0048] Figure 5 The diagram shows a schematic representation of the structure of a model training system according to an embodiment of this application;

[0049] Figure 6 The diagram shows a schematic representation of the structure of a model training device or an image recognition device according to an embodiment of this application. Detailed Implementation

[0050] Regarding the application areas of this application, artificial intelligence (AI) is a discipline that studies the use of computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing; AI software technologies include computer vision, speech recognition, natural language processing, as well as deep learning, big data processing, and knowledge graph technologies.

[0051] Computer vision is a science that studies how to make machines "see". It refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make the computer-processed images more suitable for human eyes to observe or to be transmitted to instruments for detection.

[0052] Deep learning is a new research direction in the field of machine learning. Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to recognize data such as text, images, and sound.

[0053] Self-supervised learning is a model training method that mainly utilizes auxiliary tasks to mine its own supervisory information from large-scale unsupervised data. The network is then trained using this constructed supervisory information, thereby learning representations that are valuable for downstream tasks.

[0054] In the field of deep learning for machines within the field of artificial intelligence, during the process of realizing this application, the inventors discovered that supervised learning, while achieving great success in the past, heavily relies on expensive and rare human labels and is prone to generalization errors, spurious correlations, and adversarial attacks. Therefore, research on supervised learning has reached a bottleneck. Recently, pre-training models based on self-supervised learning have made significant progress in computer vision due to their excellent data utilization efficiency and generalization ability, forming a mainstream training paradigm known as "self-supervised pre-training - task-based fine-tuning."

[0055] Against this backdrop, this application proposes a content-aware generative computer vision self-supervised learning training method to address the problems existing in supervised learning and introduces content-aware technology to optimize current generative self-supervised learning methods.

[0056] Currently, self-supervised learning can be divided into three technical directions: 1) Contrastive learning, which mainly involves: firstly constructing positive and negative examples for a batch of data through data augmentation, that is, the original samples in the batch and the samples augmented based on them are regarded as positive examples, while other samples and samples augmented based on other samples are regarded as negative examples; then, the positive and negative samples are respectively input into two neural network models with the same structure, and the two neural network models with the same structure output the embedded representations of the samples simultaneously. Finally, through a specific loss function (such as calculating the distance between the two embedded representations), the distance between positive samples is reduced and the distance between negative samples is increased, thereby enabling the neural network model to learn the ability to extract sample representations without labels, replacing the pre-training stage that previously required large-scale labeled data.

[0057] 2) Generative learning generally employs an encoder-decoder structure. The main steps are: an encoder encodes the input x into an explicit representation vector z, and then a decoder reconstructs x from the explicit representation vector z. ‘ Finally, by minimizing the reconstruction error (i.e., minimizing x and x), ‘ The difference (in the model) allows the model to learn explicit representation vectors z containing latent information even without labels. * .

[0058] 3) Generative contrastive learning, also known as adversarial learning, combines the previous two methods and adopts a "generator-discriminator" structure. The generator generally adopts an "encoder-decoder" structure, and the discriminator generally adopts a classifier structure. The main approach is to first generate fake samples through the generator, and then input the real and fake samples into the discriminator at the same time to let it distinguish between real and fake samples.

[0059] Among the mainstream algorithms in contrastive learning, the MoCo series, SimCLR, and MOBY are examples. These algorithms all suffer from the following problems: They employ a "Siamese network" training structure, meaning that each step in the training phase requires forward computation of two identical neural network models, essentially performing two training iterations, thus consuming significantly more computing power and time. Furthermore, different model parameter update strategies result in varying memory / GPU space consumption; in the worst case, both neural network models require parameter updates, doubling the memory / GPU space consumption compared to supervised training. Additionally, current contrastive learning heavily relies on data augmentation techniques. To ensure model performance, different data augmentation techniques need to be carefully designed for different tasks. Therefore, model performance is limited by the design of data augmentation techniques, and the knowledge learned by the model may only represent certain transformation characteristics rather than truly useful information. Finally, current contrastive learning typically requires a large number of negative samples during training, resulting in enormous memory / GPU space consumption.

[0060] Current generative learning methods also have some problems: the reconstruction objective determines the semantic level of the latent representation learned by the encoder; in other words, it affects the information density of the latent representation, and ultimately, the information density of the latent representation also affects the performance of downstream tasks. The reconstruction objective of the mainstream algorithms mentioned above is pixel-level restoration, that is, restoring the pixel values ​​at the corresponding positions. However, pixels are data with very low information density and high redundancy. For example, removing some pixels from an image will not affect the semantics (content) of the entire image. Therefore, the "pixel reconstruction" objective actually provides limited information to the encoder, which to some extent limits the semantic level of the latent representation.

[0061] The information carried by pixel values ​​is the content's color and brightness, and this information is extremely susceptible to noise from environmental factors, such as changes in ambient light intensity. Therefore, the latent representation learned by the encoder is also affected by noise to some extent, meaning its robustness is relatively weak.

[0062] To address the above problems, this invention proposes a content-aware generative computer vision self-supervised learning training method. This application employs a more efficient generative self-supervised learning method, effectively solving the problems inherent in supervised learning. Furthermore, addressing the shortcomings of current generative self-supervised learning methods, it introduces content-aware technology, using energy maps to quantify the importance of image content, changing the reconstruction objective from "reconstructing pixels" to "reconstructing energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness.

[0063] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0064] Example 1

[0065] Figure 1 The diagram illustrates the steps of a model training method according to an embodiment of this application.

[0066] like Figure 1 As shown, the model training method of this application embodiment includes the following steps:

[0067] S1: Obtain sample images for computer vision; i.e., RGB images.

[0068] S2: Calculate the sample energy map based on the sample image; the sample energy map includes the energy data of the sample image.

[0069] Specifically, this includes: traversing the sample image along the x-axis and the sample content along the x-axis; calculating the gradient intensity along the x-axis of the sample image according to the energy calculation formula, which is used as the energy data along the x-axis of the sample energy map; calculating the gradient intensity along the y-axis of the sample image according to the energy calculation formula, which is used as the energy data along the y-axis of the sample energy map; the data size of the energy of the sample image is directly proportional to the importance of the sample content.

[0070] S3: Input the sample image into the recognition model to obtain energy reconstruction data through prediction;

[0071] S4: Train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map.

[0072] The encoder of the recognition model in this application is an arbitrary Transformer structure model, and the decoder is an MLP regressor with a "fully connected layer + nonlinear activation function + fully connected layer" structure. The decoder output is the content-level reconstruction result of all mask blocks. The loss value in training is the supervision signal, which comes from the error value of the corresponding local block of all reconstruction results and the true energy map. Finally, the encoder and decoder model parameters are updated based on the loss value and the gradient backpropagation mechanism.

[0073] In addition, this application draws on the current mainstream generative learning MIM model, namely the mask-based image modeling method. This application provides a brief description of the mask-based image modeling steps, and for specific implementation details, please refer to the existing MIM model.

[0074] Figure 2 The diagram illustrates the steps for obtaining energy reconfiguration data according to an embodiment of this application.

[0075] Expanding on the description, such as Figure 2 As shown, in S3, the sample image is input into the model for prediction to obtain energy reconstruction data, including:

[0076] S31: Perform block segmentation, convolution, and random masking operations on the sample image to obtain local block encoding;

[0077] S32: Input local block encoding into an encoder with a Transformer structure and output encoded features with multiple representation dimensions;

[0078] S33: Input the encoded features into the MLP regressor to obtain energy reconstruction data.

[0079] Figure 3 The diagram shows a schematic representation of the model training method according to an embodiment of this application.

[0080] like Figure 3 As shown, in step S2, the sample energy map is calculated based on the sample image. Assuming the input data is an RGB image I with scale H*W, the sample energy map E needs to be calculated first using the energy calculation formula. The sample energy map can effectively reflect the important content features in the image, and the magnitude of the energy is directly proportional to the importance of the content.

[0081] The formula for calculating energy e1(I) is:

[0082]

[0083] in and These represent the gradient intensity along the x-axis and the gradient intensity along the y-axis, respectively.

[0084] Then, referring to step S31, the sample image is sliced, convolved, and randomly masked to obtain local block encoding. Specifically, this includes: dividing the sample image I into equal-scale blocks to obtain N local blocks; wherein the scale of the sample image I is H*W, the size of each local block is n*m, and the number of local blocks is...

[0085] Using convolution kernels Perform convolution operations on all local blocks to obtain N one-dimensional local block codes: Where c0 is the number of input channels of the encoder, and c0 is the number of the local block encoding.

[0086] After completing the mapping using convolution, a random masking operation is required. This involves randomly selecting a certain proportion of local block codes from N one-dimensional local block codes to form a subset of mask blocks, and then applying a normal distribution. The local block data in the mask block subset is reinitialized, ultimately resulting in the mask block subset MT = {mt0, mt1, ..., mt}. k The remaining local blocks then form a subset of visible blocks T = {t0, t1, ..., t}. N-k}

[0087] Therefore, performing a random masking operation on the local block coding yields a local block coding that includes a masked block subset and a visible block subset; where the masked block subset MT = {mt0, mt1, ..., mt} k The visible block subset T = {t0, t1, ..., t} N-k}

[0088] Figure 3 Not shown, at this point, the sample energy map is divided into equal-scale blocks, resulting in N local energy blocks. i ∈R n*m The mask block energy tag set ME = {me0, me1, ..., me} is determined by filtering the mask block subset MT based on its position. k}; label each mask block with energy. i Stretched into a one-dimensional vector me′ i ∈R (n*m)*1 This yields a new mask block energy tag set ME′={me′0,me′1,…,me′}. k The energy data serves as the final sample energy map.

[0089] like Figure 3 As shown, next, referring to step S32, the local block encoding is input into the encoder using the Transformer structure, and the output is a multi-dimensional encoding feature, including: calculating the positional encoding PE = {pe0, pe1, ..., pe1} based on the corresponding positional information of the local block encoding in the image.N Then, the position code and the local block code are added one-to-one to obtain the local block code with position information. Finally, the local block encoding with location information is input into an encoder with a Transformer structure, which outputs encoded features with multiple representation dimensions. Among them, c z For the representation dimension.

[0090] Furthermore, such as Figure 3 As shown, in step S33, the encoded features are input into the MLP regressor to obtain energy reconstruction data, including: inputting the encoded features... The normalized energy values ​​of the mask blocks are predicted by the MLP regressor, resulting in the mask block energy reconstruction result set MP = {mp0, mp1, ..., mp2}. k}, where mp i ∈R n*m .

[0091] The reconstruction objective of the MLP regressor is to reconstruct the energy value of the mask region. The MLP regressor adopts a structure of "fully connected layer + nonlinear activation function ReLU + fully connected layer", which can be defined as MLP = FC1(ReLU(FC0(Z))).

[0092] Finally, the energy reconstruction results for each mask block are processed. i By stretching, we obtain a one-dimensional vector mp′. i ∈R (n*m)*1 That is, the energy reconstruction data MP′={mp′0,mp′1,…,mp′ is obtained. k}

[0093] After obtaining the energy reconstruction data and the energy data of the sample energy map through the above operations, the error between the two is calculated, and the model is trained.

[0094] Specifically, this includes: based on the energy reconstruction data MP′={mp′0,mp′1,…,mp′ k} and the mask block energy tag set ME ′ ={me0 ′ e1 ′ ,…,e ′ k The reconstruction error is calculated based on the L2 distance.

[0095] Based on the reconstruction error l, the parameters of the encoder and MLP regressor are updated iteratively through gradient backpropagation.

[0096] This addresses the problems of poor generalization, poor robustness, and strong dependence on labeled data in current supervised learning. This application effectively utilizes a large amount of unlabeled data through a self-supervised method and mines truly effective information from a large amount of unlabeled datasets, thereby improving the generalization and robustness of the model.

[0097] This application employs a more efficient generative self-supervised learning method for model training, effectively addressing the problems inherent in supervised learning. Furthermore, to address the shortcomings of current generative self-supervised learning, content-aware technology is introduced, using energy maps to quantify the importance of image content. The reconstruction objective is changed from "reconstructing pixels" to "reconstructing energy," thereby providing the encoder with higher semantic supervision information during training. Ultimately, this improves the semantic level of the latent representations learned by the encoder and enhances feature robustness.

[0098] In summary, the model training method of this application involves acquiring sample images from computer vision; calculating a sample energy map based on the sample images; the sample energy map including energy data of the sample images; inputting the sample images into a recognition model for prediction to obtain energy reconstruction data; and training the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map. This application, by calculating the sample energy map of the sample images and using the energy map to quantify the importance of image content, changes the reconstruction target from "reconstructing pixels" to "reconstructing energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness.

[0099] Example 2

[0100] This embodiment provides a method for image recognition using a recognition model trained by any of the models in Embodiment 1. For details not disclosed in the image recognition method of this embodiment, please refer to the specific implementation of the model training method in the embodiment.

[0101] Figure 4 The figure shows a schematic diagram of the steps of an image recognition method according to an embodiment of this application.

[0102] like Figure 4 As shown, the image recognition method in this application includes:

[0103] S10: Acquire the image to be detected;

[0104] S20: Use a recognition model to predict objects in the image to be detected, so as to obtain the annotation information of each target object in the image to be detected; wherein, the recognition model is obtained by any of the above model training methods.

[0105] Example 3

[0106] This embodiment provides a model training system. For details not disclosed in this embodiment, please refer to the specific implementation of the model training methods in other embodiments.

[0107] Figure 5 The diagram shows a schematic representation of a model training system according to an embodiment of this application.

[0108] like Figure 5 As shown, the model training system of this application embodiment specifically includes a sample acquisition module 10, an energy map module 20, an energy reconstruction module 30, and a model training module 40.

[0109] Specifically,

[0110] The sample acquisition module 10 is used to acquire sample images for computer vision, namely RGB images.

[0111] The energy map module 20 is used to calculate the sample energy map based on the sample image; the sample energy map includes the energy data of the sample image.

[0112] Specifically, this includes: traversing the sample image along the x-axis and the sample content along the x-axis; calculating the gradient intensity along the x-axis of the sample image according to the energy calculation formula, which is used as the energy data along the x-axis of the sample energy map; calculating the gradient intensity along the y-axis of the sample image according to the energy calculation formula, which is used as the energy data along the y-axis of the sample energy map; the data size of the energy of the sample image is directly proportional to the importance of the sample content.

[0113] The energy reconstruction module 30 is used to input the sample image into the recognition model to predict and obtain energy reconstruction data;

[0114] The model training module 40 is used to train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map.

[0115] The encoder of the recognition model in this application is an arbitrary Transformer structure model, and the decoder is an MLP regressor with a "fully connected layer + nonlinear activation function + fully connected layer" structure. The decoder output is the content-level reconstruction result of all mask blocks. The loss value in training is the supervision signal, which comes from the error value of the corresponding local block of all reconstruction results and the true energy map. Finally, the encoder and decoder model parameters are updated based on the loss value and the gradient backpropagation mechanism.

[0116] The model training system of this application acquires computer vision sample images through a sample acquisition module 10; an energy map module 20 calculates a sample energy map based on the sample images; the sample energy map includes energy data of the sample images; an energy reconstruction module 30 inputs the sample images into a recognition model for prediction to obtain energy reconstruction data; and a model training module 40 trains the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map. This application calculates the sample energy map of the sample images, uses the energy map to quantify the importance of image content, and changes the reconstruction target from "reconstructing pixels" to "reconstructing energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness.

[0117] Example 4

[0118] This embodiment provides a model training device or an image recognition device. For details not disclosed in the model training device or image recognition device of this embodiment, please refer to the specific implementation content of the model training method or system in other embodiments.

[0119] Figure 6 The diagram shows a schematic representation of the structure of a model training device 400 or an image recognition device according to an embodiment of this application.

[0120] like Figure 6 As shown, the model training device 400 or image recognition device includes:

[0121] Memory 402: Used to store executable instructions; and

[0122] Processor 401: Used to connect with memory 402 to execute executable instructions to complete model training methods or image recognition methods.

[0123] Those skilled in the art will understand that the illustration Figure 6 This is merely an example of a model training device 400 or an image recognition device, and does not constitute a limitation on the model training device 400 or the image recognition device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the model training device 400 may also include input / output devices, network access devices, buses, etc.

[0124] The processor 401 (Central Processing Unit, CPU) can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 401 can be any conventional processor. Processor 401 is the control center of the model training device 400, connecting all parts of the model training device 400 through various interfaces and lines.

[0125] The memory 402 can be used to store computer-readable instructions. The processor 401 implements various functions of the model training device 400 by running or executing the computer-readable instructions or modules stored in the memory 402 and by calling the data stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the model training device 400, etc. In addition, the memory 402 may include a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, read-only memory (ROM), random access memory (RAM), or other non-volatile / volatile storage devices.

[0126] If the modules integrated in the model training device 400 are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of the various method embodiments described above.

[0127] Example 5

[0128] This embodiment provides a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor to implement the model training method in other embodiments.

[0129] The model training device and storage medium of this application acquire sample images from computer vision; calculate a sample energy map based on the sample images; the sample energy map includes energy data of the sample images; input the sample images into a recognition model for prediction to obtain energy reconstruction data; and train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map. This application calculates the sample energy map of the sample images, uses the energy map to quantify the importance of image content, and changes the reconstruction target from "reconstructing pixels" to "reconstructing energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0135] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0136] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0137] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A model training method, characterized in that, include: Obtain sample images for computer vision; The sample energy map is calculated based on the sample image; The sample energy map includes energy data of the sample image; The sample images are input into the recognition model to obtain energy reconstruction data through prediction. The recognition model includes an encoder and a decoder, and the output of the decoder is the energy reconstruction data. The recognition model is trained based on the error between the energy reconstruction data and the energy data of the sample energy map. The step of inputting the sample image into the model for prediction to obtain energy reconstruction data includes: The sample image is segmented, convolved, and randomly masked to obtain local block encoding; The local block encoding is input into an encoder with a Transformer structure, and the output is encoded features with multiple representation dimensions. The encoded features are input into an MLP regressor to obtain energy reconstruction data, including: The reconstruction objective of the MLP regressor is to reconstruct the energy value of the masked region, and to convert the encoded features... The normalized energy values ​​of the mask blocks are predicted by the MLP regressor, and the mask block energy reconstruction result set is obtained. ,in Where N is the number of local blocks, which are obtained by dividing the sample image into equal-scale blocks. The representation dimension of the encoded feature is denoted by , and the size of each local block is . ; Energy reconstruction results for each mask block Stretching yields a one-dimensional vector. Finally, energy reconstruction data was obtained. .

2. The model training method according to claim 1, characterized in that, The step of calculating the sample energy map based on the sample image includes: Traverse the sample image along the x-axis and the sample content along the x-axis; The gradient intensity along the x-axis of the sample image is calculated using the energy calculation formula, and is used as the energy data along the x-axis of the sample energy map. The gradient intensity along the y-axis of the sample image is also calculated using the energy calculation formula, and is used as the energy data along the y-axis of the sample energy map. The magnitude of the energy data of the sample image is directly proportional to the importance of the sample content.

3. The model training method according to claim 1, characterized in that, The step of performing block segmentation, convolution, and random masking operations on the sample image to obtain local block encoding includes: For the sample image Perform equal-scale segmentation to obtain N local blocks; among which, the sample image The scale is H*W, and the number of local blocks is ; Using convolution kernels Perform convolution operations on all local blocks to obtain One-dimensional local block encoding: ;in, The number of input channels for the encoder. The number for the local block encoding; Perform a random masking operation on the local block encoding to obtain local block encoding that includes a masked block subset and a visible block subset; wherein, the masked block subset... Visual block subset .

4. The model training method according to claim 3, characterized in that, The step of inputting the local block encoding into an encoder with a Transformer structure and outputting multi-dimensional encoded features includes: The positional code is calculated based on the corresponding positional information of the local block code in the image. ; The position code and the local block code are added one-to-one to obtain the local block code containing position information. ; The local block encoding with location information is input into an encoder with a Transformer structure, and the output is a multi-dimensional encoded feature. .

5. The model training method according to claim 4, characterized in that, The step of training the model based on the error between the energy reconstruction data and the energy data of the sample energy map includes: The sample energy map is divided into equal-scale blocks to obtain N local energy blocks. ; According to the mask block set Location filtering determines mask block energy tag set ; Label each mask block with energy Stretched into a one-dimensional vector A new set of mask block energy tags is obtained. The energy data serves as the final sample energy map. According to the energy reconstruction data and mask block energy tag set ,based on Distance calculation reconstruction error ; Based on reconstruction error The recognition model is iteratively trained and the parameters of the encoder and MLP regressor are updated through gradient backpropagation.

6. An image recognition method, characterized in that, include: Acquire the image to be detected; An identification model is used to predict objects in the image to be detected in order to obtain the annotation information of each target object in the image to be detected; wherein the identification model is obtained by the model training method according to any one of claims 1-5.

7. A model training system, characterized in that, include: Sample acquisition module: Used to acquire sample images for computer vision; Energy map module: used to calculate the sample energy map based on the sample image; The sample energy map includes energy data of the sample image; Energy reconstruction module: used to input the sample image into the recognition model to obtain energy reconstruction data through prediction; The recognition model includes an encoder and a decoder, and the output of the decoder is the energy reconstruction data. Model training module: used to train the recognition model based on the error between the energy reconstruction data and the energy data of the sample energy map; The energy reconfiguration module is also used for: The sample image is segmented, convolved, and randomly masked to obtain local block encoding; The local block encoding is input into an encoder with a Transformer structure, and the output is encoded features with multiple representation dimensions. The encoded features are input into an MLP regressor to obtain energy reconstruction data, including: The reconstruction objective of the MLP regressor is to reconstruct the energy value of the masked region, and to convert the encoded features... The normalized energy values ​​of the mask blocks are predicted by the MLP regressor, and the mask block energy reconstruction result set is obtained. ,in Where N is the number of local blocks, which are obtained by dividing the sample image into equal-scale blocks. The representation dimension of the encoded feature is denoted by , and the size of each local block is . ; Energy reconstruction results for each mask block Stretching yields a one-dimensional vector. Finally, energy reconstruction data was obtained. .

8. A computer-readable storage medium, characterized in that, It stores a computer program; the computer program is executed by a processor to implement the model training method as described in any one of claims 1-5 and the image recognition method as described in claim 6.

Images