A network model generation method, device, terminal and storage medium

Abstract

The present application is applicable to the computer field, and provides a method, device, terminal and storage medium for generating a network model. The method for generating the above network model includes: obtaining multiple sample image groups, each sample image group includes a first sample image and a second sample image; using multiple sample image groups, training the network model to be trained, and obtaining an initial network model that can accurately output the first classification confidence corresponding to each sample image group; reconstructing the output layer of the initial network model to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified. The embodiments of the present application can improve the generation efficiency of the end-to-end model.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device, terminal and storage medium for generating a network model. Background Art

[0002] The use of network models for image processing has good practicality. When classifying and identifying two images, the network model can be used to process the two input images to obtain the classification results of the two images.

[0003] Currently, the complexity of the end-to-end network model training process is relatively high. Therefore, a network model generation method is needed to obtain an end-to-end network model while reducing the complexity of the training process. Summary of the invention

[0004] The embodiments of the present application provide a method, device, terminal and storage medium for generating a network model, which can reduce the complexity of the end-to-end network model training process while obtaining an end-to-end network model.

[0005] A first aspect of an embodiment of the present application provides a method for generating a network model, including:

[0006] Acquire multiple sample image groups, each sample image group comprising a first sample image and a second sample image;

[0007] Using the multiple sample image groups, the network model to be trained is trained to obtain an initial network model that can accurately output a first classification confidence corresponding to each sample image group; the first classification confidence represents the degree of similarity or difference between the first sample image and the second sample image in the corresponding sample image group;

[0008] The output layer of the initial network model is reconstructed to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified

[0009] A second aspect of an embodiment of the present application provides a method for identifying a device status, including:

[0010] Acquire a first group of images to be classified, wherein the first group of images to be classified includes a reference image and an image to be identified of a device;

[0011] Input the first image group to be classified into the device state recognition network model, and obtain a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified output by the device state recognition network model; wherein the device state recognition network model is the target network model provided in the first aspect of the embodiment of the present application;

[0012] A target device state of the device is determined based on the first change detection heat map and / or the second classification confidence, and a reference device state corresponding to the benchmark image.

[0013] A third aspect of an embodiment of the present application provides a device for generating a network model, including:

[0014] A first acquisition unit, used to acquire a plurality of sample image groups, each sample image group comprising a first sample image and a second sample image;

[0015] A network model training unit, configured to train a network model to be trained using the plurality of sample image groups, to obtain an initial network model capable of accurately outputting a first classification confidence corresponding to each sample image group; the first classification confidence represents a degree of similarity or difference between the first sample image and the second sample image in the corresponding sample image group;

[0016] A network model generation unit is used to reconstruct the output layer of the initial network model to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified.

[0017] A fourth aspect of an embodiment of the present application provides a device for identifying a device status, including:

[0018] A second acquisition unit is used to acquire a first group of images to be classified, wherein the first group of images to be classified includes a reference image and an image to be identified of the device;

[0019] a third acquisition unit, configured to input the first image group to be classified into a device state recognition network model, and obtain a first change detection heat map corresponding to the first image group to be classified output by the device state recognition network model, and a second classification confidence corresponding to the first image group to be classified; wherein the device state recognition network model is the target network model provided in the first aspect of the embodiment of the present application;

[0020] A device state identification unit is used to determine a target device state of the device based on the first change detection heat map and / or the second classification confidence, and a reference device state corresponding to the benchmark image.

[0021] A fifth aspect of an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0022] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0023] A seventh aspect of an embodiment of the present application provides a computer program product, which, when executed on a terminal, enables the terminal to implement the steps of the method.

[0024] In an implementation manner of the present application, a non-end-to-end network model to be trained is first trained to obtain a trained non-end-to-end initial network model, and then the output layer of the initial network model is reconstructed to obtain a trained end-to-end target network model. On the one hand, by training in the training method of a non-end-to-end network model, there is no need to perform complex image processing when judging convergence like an end-to-end network model, which reduces the complexity of training and improves the training efficiency of the end-to-end model. On the other hand, the target network model finally obtained is an end-to-end network model, which improves the practicability of the target network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 It is a schematic diagram of an implementation flow of a method for generating a network model provided in an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of a specific implementation flow of step S101 provided in an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a specific implementation flow of step S102 provided in an embodiment of the present application;

[0029] Figure 4It is a schematic diagram of a first specific implementation process of generating a second change detection heat map provided in an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of a second specific implementation process of generating a second change detection heat map provided in an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of the initial network model provided in the embodiment of the present application;

[0032] Figure 7 It is a schematic diagram of an implementation flow of a method for identifying a device status provided in an embodiment of the present application;

[0033] Figure 8 This is a schematic diagram of the effect of identifying the device status provided by an embodiment of the present application;

[0034] Fig. 9 It is a structural schematic diagram of a network model generation device provided in an embodiment of the present application;

[0035] Fig.10 It is a structural schematic diagram of a device status identification apparatus provided in an embodiment of the present application;

[0036] Fig.11 It is a schematic diagram of the structure of the terminal provided in the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0038] The use of network models for image processing has good practicality. When classifying and identifying two images, the network model can be used to process the two input images to obtain the classification results of the two images.

[0039] For example, during the inspection of industrial scenes, it is often necessary to record the normal status of certain specific equipment or areas and determine whether the status of the equipment has changed during the inspection. At this time, the network model can be used to process the input image to be identified and the reference image, output the similarity information or difference information between the two images, and then derive the device status of the equipment.

[0040] Currently, the complexity of the end-to-end network model training process is relatively high. Therefore, the present application proposes a method for generating a network model, which reduces the complexity of the training process while obtaining an end-to-end network model.

[0041] In some embodiments of the present application, the end-to-end network model obtained based on the above generation method can be applied to inspection scenarios. The input image to be identified and the reference image are processed using the generated network model, and the terminal can obtain the change detection heat map and classification confidence between the two images output by the generated network model, and then realize the identification of the equipment status according to the change detection heat map and classification confidence.

[0042] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0043] Figure 1 The present invention provides a schematic diagram of a method for generating a network model according to an embodiment of the present invention, which can be applied to a terminal and can be used to reduce the complexity of training the network model while generating an end-to-end network model. The terminal can be a server, a computer or other device.

[0044] Specifically, the above network model generation method may include the following steps S101 to S103.

[0045] Step S101, obtaining multiple sample image groups.

[0046] In an embodiment of the present application, the sample image groups are used to train the network model to be trained, and each sample image group includes a first sample image and a second sample image.

[0047] The existing network model training method generally requires the use of balanced positive and negative samples for model training. When the trained network model is finally applied to device status recognition, the positive samples and negative samples correspond to sample images under normal device status and sample images under abnormal device status, respectively. In actual applications, sample images under abnormal status can only be collected under specific circumstances. In other words, in the process of sample image collection, only images corresponding to normal device status can be obtained.

[0048] Based on this, in some embodiments of the present application, Figure 2 As shown, the acquisition of the sample image group may include the following steps S201 to S203.

[0049] Step S201: Acquire a first sample image obtained by photographing a target object from multiple points.

[0050] The target object can be adjusted according to the specific application scenario of the network model after training. For example, when the network model is applied to device status recognition after training, the target object can be the device whose device status needs to be recognized.

[0051] The above-mentioned points refer to the positions at which the target object is photographed. In some embodiments of the present application, different first sample images can be obtained by photographing the target object at different points, and the image contents of the first sample images corresponding to different points are different.

[0052] Step S202 : performing noise enhancement on a first sample image corresponding to each of the plurality of points to obtain a second sample image corresponding to each of the plurality of points.

[0053] Specifically, the above-mentioned noise enhancement refers to introducing interference factors such as nonlinear light, noise, brightness changes, etc. to simulate environmental changes when acquiring images, while ensuring that the specific content of the image is not modified, so as to ensure that the enhanced second sample image does not have any change area of ​​interest relative to the corresponding first sample image.

[0054] Step S203 , selecting two first target sample images from the first sample image and the second sample image as a positive sample image group, and selecting two second target sample images from the first sample image and the second sample image as a negative sample image group, to obtain a plurality of sample image groups.

[0055] The points corresponding to the two first target sample images are the same, and the points corresponding to the two second target sample images are different.

[0056] That is to say, the present application selects any two sample images from the first sample image and the second sample image at the same point position to form a positive sample image group. And the first point position of two different points corresponds to the first sample image or the second sample image, and the second point position corresponds to the first sample image or the second sample image to form a negative sample image group. By analogy, multiple positive sample image groups and multiple negative sample image groups can be obtained. Each image group in the multiple positive sample image groups and the multiple negative sample image groups is respectively regarded as a sample image group, and finally multiple sample image groups can be obtained.

[0057] In order to make the final trained network model more universal, in some embodiments of the present application, the more the above-mentioned points are, the better, and the more the number of different sample image groups is, the better. At the same time, the ratio of positive sample image groups to negative sample image groups in the multiple sample image groups obtained should be relatively balanced.

[0058] In the implementation manner of the present application, by grouping sample images corresponding to the same point into a positive sample image group, and grouping sample images corresponding to different points into a positive sample image group, on the one hand, in practical applications, there is no need to specifically acquire corresponding images under different states. For example, when the method provided in the present application is applied to equipment status detection, there is no need to specifically acquire images taken of equipment in an abnormal state, which makes sample collection more convenient and more practical. At the same time, there is no need to manually calibrate the image to determine the positive and negative samples. On the other hand, in practical applications, images taken at different points can be accurately identified. When the method provided in the present application is applied to inspection, the consistency of the points of the image to be identified and the reference image can be guaranteed.

[0059] In addition, the present application can avoid interference factors such as nonlinear illumination changes from affecting the accuracy of the trained network model through noise enhancement, thereby improving the accuracy and universality of the network model.

[0060] Step S102, using multiple sample image groups, training the network model to be trained to obtain an initial network model that can accurately output the first classification confidence corresponding to each sample image group.

[0061] The above classification confidence represents the similarity or difference between the two images. The first classification confidence is the classification confidence between the first sample image and the second sample image output by the initial network model.

[0062] That is, in an embodiment of the present application, based on multiple sample image groups, the network model to be trained can be trained to converge the network model to obtain an initial network model. The initial network model can output an accurate first classification confidence between the first sample image and the second sample image based on the first sample image and the second sample image contained in a single sample image group.

[0063] Specifically, in some embodiments of the present application, Figure 3 As shown, the training of the network model to be trained may include the following steps S301 to S303.

[0064] Step S301: Select a target training sample from a sample image set.

[0065] Each target training sample is associated with a sample positive or negative property, which is used to identify whether the target training sample belongs to a positive sample image group or a negative sample image group.

[0066] Step S302, input each target training sample into the network model to be trained respectively, obtain the third classification confidence corresponding to the target training sample output by the network model to be trained, and count the accuracy of the network model to be trained according to the third classification confidence corresponding to each target training sample and the sample positivity associated with each target training sample.

[0067] In some embodiments of the present application, the above-mentioned third classification confidence can be a confidence value output by a binary classifier, wherein, when the third classification confidence is 0, it indicates that the two images in the target training sample are the same; when the third classification confidence is 1, it indicates that the two images in the target training sample are not the same.

[0068] Specifically, in some embodiments of the present application, the first target training sample can be input into the network model to be trained, and the third classification confidence corresponding to the first target training sample output by the network model to be trained is obtained. According to the sample positivity associated with the first target training sample and the third classification confidence corresponding to the first target training sample, it can be determined whether the output is accurate. That is, when the target training sample belongs to the positive sample image group and the third classification confidence is 0, or the target training sample belongs to the negative sample image group and the third classification confidence is 1, then the output is accurate; conversely, when the target training sample belongs to the positive sample image group and the third classification confidence is 1, or the target training sample belongs to the negative sample image group and the third classification confidence is 0, then the output is inaccurate. By analogy, after performing the above processing on each target training sample, the accuracy of the network model to be trained can be statistically obtained.

[0069] Step S303, if the accuracy of the network model to be trained is less than the accuracy threshold, adjust the parameters in the network model to be trained, and re-execute the step of selecting the target training sample from the sample image set and the subsequent steps until the accuracy of the network model to be trained is greater than or equal to the accuracy threshold, and the initial network model is obtained.

[0070] The accuracy threshold refers to the minimum accuracy required for the network model to be trained when the network model can output accurate classification confidence.

[0071] In some embodiments of the present application, if the accuracy of the network model to be trained is less than the accuracy threshold, it means that the network model to be trained has not converged and cannot output accurate classification confidence. It is necessary to adjust the parameters in the network model to be trained, and re-execute the step of selecting target training samples from the sample image set and subsequent steps until the accuracy of the network model to be trained is greater than or equal to the accuracy threshold, indicating that the network model to be trained has converged and can output accurate classification confidence, thereby obtaining the initial network model.

[0072] In the implementation manner of the present application, by training the network model to be trained using the target training samples and the sample positivity of the target training samples, an initial network model that can accurately output the first classification confidence corresponding to each sample image group can be obtained.

[0073] Step S103, reconstructing the output layer of the initial network model to obtain the target network model.

[0074] The target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified.

[0075] The first change detection heat map is used to represent the difference between two images in the first set of images to be classified. The second classification confidence is the classification confidence between the two images in the set of images to be classified output by the target network model.

[0076] In an implementation manner of the present application, a non-end-to-end network model to be trained is first trained to obtain a trained non-end-to-end initial network model, and then the output layer of the initial network model is reconstructed to obtain a trained end-to-end target network model. On the one hand, by training in the training method of a non-end-to-end network model, there is no need to perform complex image processing when judging convergence like an end-to-end network model, which reduces the complexity of training and improves the training efficiency of the end-to-end model. On the other hand, the target network model finally obtained is an end-to-end network model, which improves the practicability of the target network model.

[0077] Specifically, in some embodiments of the present application, the initial network model includes a change detection heat map generation layer and an output layer.

[0078] Among them, the above-mentioned change detection heat map generation layer is used to process the input second image group to be classified, and output the second change detection heat map corresponding to the second image group to be classified; the above-mentioned output layer is used to classify according to the second change detection heat map to obtain the first classification confidence corresponding to the second image group to be classified.

[0079] Specifically, in some embodiments of the present application, Figure 4 As shown, the specific execution steps of the above-mentioned change detection heat map generation layer include the following steps S401 to S404.

[0080] Step S401 : performing feature extraction on two images to be classified contained in the second group of images to be classified, respectively, to obtain a plurality of first feature maps corresponding to the two images to be classified.

[0081] Among them, the sizes of multiple first feature maps corresponding to the same image to be classified are different.

[0082] Specifically, in some embodiments of the present application, assuming that the two images to be classified are respectively a first image to be classified and a second image to be classified, a feature extraction network can be used to perform feature extraction of different scales on the first image to be classified to obtain first feature maps of different sizes corresponding to the first image to be classified; and then the same feature extraction network can be used to perform feature extraction of different scales on the second image to be classified to obtain first feature maps of different sizes corresponding to the second image to be classified.

[0083] Step S402 , up-sampling the plurality of first feature maps corresponding to the two images to be classified respectively, to obtain a second feature map corresponding to each first feature map.

[0084] The sizes of the second feature maps corresponding to each first feature map are the same.

[0085] In some embodiments of the present application, in order to fuse feature maps of various scales, upsampling may be performed to obtain a second feature map of the same size corresponding to each first feature map.

[0086] Step S403: splicing the second feature maps corresponding to the same image to be classified in the feature channel direction to obtain a third feature map.

[0087] That is, in the channel direction, the second feature maps corresponding to the first image to be classified can be spliced ​​to obtain the third feature map containing various scale features corresponding to the first image to be classified; similarly, in the channel direction, the second feature maps corresponding to the second image to be classified can be spliced ​​to obtain the third feature map containing various scale features corresponding to the second image to be classified.

[0088] Step S404: Generate a second change detection heat map corresponding to the second group of images to be classified according to the third feature maps respectively corresponding to the two images to be classified.

[0089] Specifically, Figure 5 As shown, in some implementations of the present application, the above step S404 may include the following steps S501 to S503.

[0090] Step S501, subtracting the third feature maps corresponding to the two images to be classified to obtain a difference map between the third feature maps corresponding to the two images to be classified.

[0091] At this time, the above difference map can roughly reflect the difference between the corresponding areas of the two images to be classified.

[0092] Step S502: Use the attention mechanism to assign different weights to each feature channel in the difference map.

[0093] In some embodiments of the present application, an attention mechanism may be used to assign higher weights to feature channels of interest in the difference map, and to assign lower weights to feature channels of no interest in the difference map.

[0094] Step S503, according to the weight of each feature channel in the difference map, the difference map is fused in the feature channel direction to obtain a difference fusion map, and the difference fusion map is normalized to obtain a second change detection heat map corresponding to the second image group to be classified.

[0095] Specifically, in some embodiments of the present application, the sum of squares can be calculated in the channel direction, and then normalized by image standardization and sigmoid activation function to form a second change detection heat map that is consistent with the size of the image to be classified. In the obtained second change detection heat map, the closer the pixel value of the pixel area is to 1, the greater the difference between the two images to be classified in the pixel area, and the closer the pixel value of the pixel area is to 0, the smaller the difference between the two images to be classified in the pixel area.

[0096] In an embodiment of the present application, by performing feature extraction, feature map splicing, subtraction, fusion and normalization on two images to be classified, a second change detection heat map for representing the degree of difference or similarity between the two images to be classified can be generated.

[0097] Considering that if the two images to be classified in the second group of images to be classified originate from the same point, the two images to be classified have a very high similarity, and the pixel values ​​on the second change detection heat map are almost close to 0. On the contrary, if the two images to be classified in the second group of images to be classified originate from different points, it is difficult to encounter exactly the same corresponding images between the two images to be classified in actual collection. Therefore, the pixel values ​​on the second change detection heat map tend to 1. In some embodiments of the present application, the change detection heat map can be flattened and a binary classifier based on numerical values ​​can be connected in series, so that the output layer of the above-mentioned network model can classify the second change detection heat map to obtain the first classification confidence corresponding to the second group of images to be classified, which is equivalent to introducing a signal that can be used for supervised learning, and at the same time directly outputs the similarity between the two images in the second group of images to be classified.

[0098] Figure 6 The structure of an initial network model provided in an embodiment of the present application is shown; wherein, image A and image B correspond to two images, such as the first sample image and the second sample image in the sample image group, respectively. After feature extraction, feature map splicing, subtraction, fusion and normalization, a second change detection heat map for representing the degree of difference or similarity between the two images can be generated. After flattening and classification by a binary classifier, the first classification confidence between the two images can be obtained.

[0099] In some implementations of the present application, the above target network model may be applied to the identification of device status.

[0100] Specifically, Figure 7 The present invention provides a schematic diagram of the implementation process of a method for identifying the status of a device provided in an embodiment of the present invention, which can be applied to a terminal and can be applied to situations where the status of a device needs to be efficiently identified. The terminal can be a computer, a server, an inspection robot, and other devices.

[0101] Specifically, the above-mentioned method for identifying the device status may include the following steps S701 to S703.

[0102] Step S701: obtaining a first group of images to be classified.

[0103] In an embodiment of the present application, the first group of images to be classified includes reference images and images of devices to be identified.

[0104] The image to be identified may be an image obtained by the inspection robot photographing the device during the inspection process; and the reference image may be a pre-stored image obtained by photographing the device in a normal state.

[0105] In some embodiments of the present application, since the inspection image often has a positional offset relative to the reference image, the terminal may further perform image registration on the image to be recognized and the reference image after acquiring the image to be recognized, so as to achieve higher recognition accuracy.

[0106] Step S702: input the first image group to be classified into the device state recognition network model, and obtain the first change detection heat map corresponding to the first image group to be classified output by the device state recognition network model, and the second classification confidence corresponding to the first image group to be classified.

[0107] Among them, the above equipment status recognition network model is Figures 1 to 6 The target network model is shown.

[0108] That is to say, in some embodiments of the present application, it can be based on Figures 1 to 6 The network model generation method shown generates a device state recognition network model, and then inputs the first image group to be classified into the trained device state recognition network model to obtain a first change detection heat map corresponding to the first image group to be classified output by the device state recognition network model, and a second classification confidence corresponding to the first image group to be classified.

[0109] Among them, the first change detection heat map and the second classification confidence can both represent the similarity or difference between the reference image and the image to be identified by the device.

[0110] Step S703: Determine the target device state of the device according to the first change detection heat map and / or the second classification confidence, and the reference device state corresponding to the benchmark image.

[0111] Specifically, in some embodiments of the present application, according to the second classification confidence, if the second classification confidence is 0, it means that the image to be identified and the reference image are similar images, based on which it can be confirmed that the target device state of the device in the image to be identified is the same as the reference device state corresponding to the reference image. If the second classification confidence is 1, it means that the image to be identified and the reference image are dissimilar images, based on which it can be confirmed that the target device state of the device in the image to be identified is different from the reference device state corresponding to the reference image.

[0112] Similarly, since the image content of the first change detection heat map represents the difference between the image to be identified and the reference image, it is also possible to determine whether the image to be identified and the reference image are similar based on the first change detection heat map, thereby determining the target device state of the device in the image to be identified based on the reference device state corresponding to the reference image.

[0113] It should be noted that the above-mentioned first change detection heat map can intuitively show the difference between the image to be identified and the reference image, and based on further image processing of the first change detection heat map, more image information can be determined. For example, the difference between the corresponding points of the image to be identified and the corresponding points of the reference image can be determined based on the first change detection heat map, which is more practical.

[0114] Figure 8 The effect of device state recognition using the device state recognition method of the present application is shown; wherein 801 is a reference image, 802 is an image to be recognized, 803 is a reference image after image registration, 804 is an image to be recognized after image registration, and 805 is a first change detection heat map output by the device state recognition network model.

[0115] In the implementation mode of the present application, by Figures 1 to 6 The network model generation method shown generates a device state recognition network model, and outputs a first change detection heat map and a second classification confidence between a reference image and an image to be recognized based on the device state recognition network model, thereby determining the target device state of the device based on the first change detection heat map and / or the second classification confidence, and the reference device state corresponding to the reference image, thereby improving the practicality, accuracy and recognition efficiency of device state recognition.

[0116] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0117] like Fig. 9 The figure is a schematic diagram of the structure of a network model generation device 900 provided in an embodiment of the present application, wherein the network model generation device 900 is configured on a terminal.

[0118] Specifically, the network model generating device 900 may include:

[0119] A first acquisition unit 901 is used to acquire a plurality of sample image groups, each sample image group comprising a first sample image and a second sample image;

[0120] A network model training unit 902 is used to train the network model to be trained using the multiple sample image groups to obtain an initial network model that can accurately output a first classification confidence corresponding to each sample image group; the first classification confidence represents the degree of similarity or difference between the first sample image and the second sample image in the corresponding sample image group;

[0121] The network model generation unit 903 is used to reconstruct the output layer of the initial network model to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified.

[0122] In some embodiments of the present application, the above-mentioned first acquisition unit 901 can also be specifically used to: acquire a first sample image obtained by photographing the target object from multiple points; perform noise enhancement on the first sample image corresponding to each of the multiple points to obtain a second sample image corresponding to each of the multiple points; select two first target sample images from the first sample image and the second sample image as a positive sample image group, and select two second target sample images from the first sample image and the second sample image as a negative sample image group to obtain the multiple sample image groups, wherein the points corresponding to the two first target sample images are the same, and the points corresponding to the two second target sample images are different.

[0123] In some embodiments of the present application, the above-mentioned network model training unit 902 can also be specifically used to: select target training samples from the sample image set, wherein each target training sample is associated with a sample positivity, and the sample positivity is used to identify that the target training sample belongs to the positive sample image group, or that the target training sample belongs to the negative sample image group; input each target training sample into the network model to be trained respectively, obtain the third classification confidence corresponding to the target training sample output by the network model to be trained, and according to the third classification confidence corresponding to each target training sample and the sample positivity associated with each target training sample, count the accuracy of the network model to be trained; if the accuracy of the network model to be trained is less than the accuracy threshold, adjust the parameters in the network model to be trained, and re-execute the step of selecting the target training sample from the sample image set and subsequent steps until the accuracy of the network model to be trained is greater than or equal to the accuracy threshold, and the initial network model is obtained.

[0124] In some embodiments of the present application, the above-mentioned initial network model includes a change detection heat map generation layer and an output layer; the change detection heat map generation layer is used to process the input second image group to be classified, and output a second change detection heat map corresponding to the second image group to be classified; the output layer is used to classify the second change detection heat map to obtain a first classification confidence corresponding to the second image group to be classified.

[0125] In some embodiments of the present application, the above-mentioned change detection heat map generation layer is also specifically used to: perform feature extraction on the two images to be classified contained in the second image group to be classified, respectively, to obtain multiple first feature maps corresponding to the two images to be classified, wherein the sizes of the multiple first feature maps corresponding to the same image to be classified are different; upsample the multiple first feature maps corresponding to the two images to be classified, respectively, to obtain a second feature map corresponding to each of the first feature maps; splice the second feature map corresponding to the same image to be classified in the feature channel direction to obtain a third feature map; generate a second change detection heat map corresponding to the second image group to be classified according to the third feature maps corresponding to the two images to be classified.

[0126] In some embodiments of the present application, the above-mentioned change detection heat map generation layer is also specifically used to: subtract the third feature maps corresponding to the two images to be classified, so as to obtain a difference map between the third feature maps corresponding to the two images to be classified; use the attention mechanism to assign different weights to each feature channel in the difference map; according to the weight of each feature channel in the difference map, fuse the difference map in the direction of the feature channel to obtain a difference fusion map, and normalize the difference fusion map to obtain a second change detection heat map corresponding to the second group of images to be classified.

[0127] It should be noted that, for the convenience and simplicity of description, the specific working process of the network model generation device 900 can refer to Figures 1 to 6 The corresponding process of the method will not be repeated here.

[0128] like Fig.10 FIG. 1 is a schematic diagram of the structure of a device status identification apparatus 1000 provided in an embodiment of the present application, wherein the device status identification apparatus 1000 is configured on a terminal.

[0129] Specifically, the device status identification apparatus 1000 may include:

[0130] The second acquisition unit 1001 is used to acquire a first group of images to be classified, where the first group of images to be classified includes a reference image and an image to be identified of a device;

[0131] The third acquisition unit 1002 is used to input the first image group to be classified into the device state recognition network model, and obtain the first change detection heat map corresponding to the first image group to be classified output by the device state recognition network model, and the second classification confidence corresponding to the first image group to be classified; wherein the device state recognition network model is Figures 1 to 6 The target network model;

[0132] The device state recognition unit 1003 is used to determine the target device state of the device according to the first change detection heat map and / or the second classification confidence, and the reference device state corresponding to the benchmark image.

[0133] It should be noted that, for the convenience and simplicity of description, the specific working process of the device status identification device 1000 can refer to Figures 7 and 8 The corresponding process of the method will not be repeated here.

[0134] like Fig.11The figure is a schematic diagram of a terminal provided in an embodiment of the present application. The terminal 11 may include: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110, such as a network model generation program. When the processor 110 executes the computer program 112, the steps in the above-mentioned embodiments of the generation method of each network model are implemented, such as Figure 1 Alternatively, when the processor 110 executes the computer program 112, the steps in the above-mentioned device status identification method embodiments are implemented, for example Figure 7 Steps S701 to S703 are shown.

[0135] Alternatively, when the processor 110 executes the computer program 112, the functions of the modules / units in the above-mentioned device embodiments are implemented, for example Fig. 9 The first acquisition unit 901, the network model training unit 902, and the network model generation unit 903 are shown. Alternatively, when the processor 110 executes the computer program 112, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig.10 The second acquisition unit 1001, the third acquisition unit 1002, and the device state identification unit 1003 are shown.

[0136] The computer program may be divided into one or more modules / units, which are stored in the memory 111 and executed by the processor 110 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the terminal.

[0137] For example, the computer program can be divided into: a first acquisition unit, a network model training unit, and a network model generation unit. The specific functions of each unit are as follows: a first acquisition unit, used to acquire multiple sample image groups, each sample image group includes a first sample image and a second sample image; a network model training unit, used to train the network model to be trained using the multiple sample image groups, and obtain an initial network model that can accurately output the first classification confidence corresponding to each sample image group; the first classification confidence represents the degree of similarity or difference between the first sample image and the second sample image in the corresponding sample image group; a network model generation unit, used to reconstruct the output layer of the initial network model to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified.

[0138] For another example, the computer program can be divided into: a second acquisition unit, a third acquisition unit, and a device state recognition unit. The specific functions of each unit are as follows: the second acquisition unit is used to acquire a first group of images to be classified, wherein the first group of images to be classified includes a reference image and an image of a device to be recognized; the third acquisition unit is used to input the first group of images to be classified into a device state recognition network model, and acquire a first change detection heat map corresponding to the first group of images to be classified output by the device state recognition network model, and a second classification confidence corresponding to the first group of images to be classified; wherein the device state recognition network model is Figures 1 to 6 The target network model; a device state identification unit, used to determine the target device state of the device based on the first change detection heat map and / or the second classification confidence, and the reference device state corresponding to the benchmark image.

[0139] The terminal may include, but is not limited to, a processor 110 and a memory 111. Those skilled in the art will appreciate that Fig.11 It is only an example of a terminal and does not constitute a limitation on the terminal. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0140] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0141] The memory 111 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. The memory 111 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 111 may also include both an internal storage unit of the terminal and an external storage device. The memory 111 is used to store the computer program and other programs and data required by the terminal. The memory 111 may also be used to temporarily store data that has been output or is to be output.

[0142] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0149] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a network model, characterized in that: include: Acquire multiple sample image groups, each sample image group comprising a first sample image and a second sample image; Using the multiple sample image groups, training the network model to be trained to obtain an initial network model that can accurately output the first classification confidence corresponding to each sample image group; The first classification confidence represents the similarity or difference between the first sample image and the second sample image in the corresponding sample image group; Reconstructing the output layer of the initial network model to obtain a target network model; the target network model is used to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified; The obtaining of the plurality of sample image groups comprises: obtaining a first sample image obtained by photographing the target object from a plurality of points; performing noise enhancement on the first sample image corresponding to each of the plurality of points, to obtain a second sample image corresponding to each of the plurality of points; selecting two first target sample images from the first sample image and the second sample image as a positive sample image group, and selecting two second target sample images from the first sample image and the second sample image as a negative sample image group, to obtain the plurality of sample image groups, wherein the points corresponding to the two first target sample images are the same, and the points corresponding to the two second target sample images are different; The method uses the multiple sample image groups to train the network model to be trained to obtain an initial network model that can accurately output the first classification confidence corresponding to each sample image group, including: selecting target training samples from the sample image set, wherein each target training sample is associated with a sample positivity, and the sample positivity is used to identify that the target training sample belongs to the positive sample image group, or that the target training sample belongs to the negative sample image group; inputting each target training sample into the network model to be trained respectively, obtaining the third classification confidence corresponding to the target training sample output by the network model to be trained, and according to the third classification confidence corresponding to each target training sample and the sample positivity associated with each target training sample, calculating the accuracy of the network model to be trained; if the accuracy of the network model to be trained is less than the accuracy threshold, adjusting the parameters in the network model to be trained, and re-executing the step of selecting the target training sample from the sample image set and subsequent steps until the accuracy of the network model to be trained is greater than or equal to the accuracy threshold, thereby obtaining the initial network model.

2. The method for generating a network model according to claim 1, wherein: The initial network model includes a change detection heat map generation layer and an output layer; The change detection heat map generation layer is used to process the input second image group to be classified and output a second change detection heat map corresponding to the second image group to be classified; The output layer is used to classify the second change detection heat map to obtain a first classification confidence corresponding to the second group of images to be classified.

3. The method for generating a network model according to claim 2, wherein: The change detection heat map generation layer is also specifically used for: Performing feature extraction on two images to be classified contained in the second group of images to be classified to obtain a plurality of first feature maps corresponding to the two images to be classified, wherein the sizes of the plurality of first feature maps corresponding to the same image to be classified are different; Upsampling the plurality of first feature maps respectively corresponding to the two images to be classified to obtain a second feature map corresponding to each of the first feature maps; splicing the second feature maps corresponding to the same image to be classified in the feature channel direction to obtain a third feature map; A second change detection heat map corresponding to the second group of images to be classified is generated according to the third feature maps respectively corresponding to the two images to be classified.

4. The method for generating a network model according to claim 3, characterized in that: The step of generating a second change detection heat map corresponding to the second group of images to be classified according to the third feature maps respectively corresponding to the two images to be classified comprises: Subtracting the third feature maps respectively corresponding to the two images to be classified to obtain a difference map between the third feature maps respectively corresponding to the two images to be classified; Using an attention mechanism to assign different weights to each feature channel in the difference map; According to the weight of each feature channel in the difference map, the difference map is fused in the feature channel direction to obtain a difference fusion map, and the difference fusion map is normalized to obtain a second change detection heat map corresponding to the second image group to be classified.

5. A method for identifying a device status, characterized in that: include: Acquire a first group of images to be classified, wherein the first group of images to be classified includes a reference image and an image to be identified of a device; Input the first image group to be classified into a device state recognition network model, and obtain a first change detection heat map corresponding to the first image group to be classified output by the device state recognition network model, and a second classification confidence corresponding to the first image group to be classified; wherein the device state recognition network model is the target network model according to any one of claims 1 to 4; A target device state of the device is determined based on the first change detection heat map and / or the second classification confidence, and a reference device state corresponding to the benchmark image.

6. A network model generation device, characterized in that: include: A first acquisition unit, used to acquire a plurality of sample image groups, each sample image group comprising a first sample image and a second sample image; A network model training unit, used to train the network model to be trained using the multiple sample image groups, to obtain an initial network model that can accurately output the first classification confidence corresponding to each sample image group; The first classification confidence represents the similarity or difference between the first sample image and the second sample image in the corresponding sample image group; A network model generation unit, configured to reconstruct the output layer of the initial network model to obtain a target network model; the target network model is configured to process the input first image group to be classified, and output a first change detection heat map corresponding to the first image group to be classified and a second classification confidence corresponding to the first image group to be classified; The first acquisition unit is used to acquire a first sample image obtained by photographing the target object from multiple points; Performing noise enhancement on the first sample image corresponding to each of the multiple points to obtain a second sample image corresponding to each of the multiple points; Selecting two first target sample images from the first sample image and the second sample image as a positive sample image group, and selecting two second target sample images from the first sample image and the second sample image as a negative sample image group, to obtain the plurality of sample image groups, wherein the points corresponding to the two first target sample images are the same, and the points corresponding to the two second target sample images are different; The network model training unit is used to: select target training samples from a sample image set, wherein each target training sample is associated with sample positivity, and the sample positivity is used to identify that the target training sample belongs to a positive sample image group, or that the target training sample belongs to a negative sample image group; input each target training sample into the network model to be trained respectively, obtain a third classification confidence corresponding to the target training sample output by the network model to be trained, and calculate the accuracy of the network model to be trained based on the third classification confidence corresponding to each target training sample and the sample positivity associated with each target training sample; if the accuracy of the network model to be trained is less than an accuracy threshold, adjust the parameters in the network model to be trained, and re-execute the step of selecting the target training sample from the sample image set and subsequent steps until the accuracy of the network model to be trained is greater than or equal to the accuracy threshold, thereby obtaining an initial network model.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented; or, when the processor executes the computer program, the steps of the method according to claim 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented; or, when the computer program is executed by a processor, the steps of the method according to claim 5 are implemented.

Images