Image processing method and device, computer readable storage medium and electronic device

Abstract

The embodiments of the present disclosure provide an image processing method and device, a computer-readable storage medium, and an electronic device. The method includes: obtaining an original image to be tested; encoding the original image to be tested to generate a target feature map of the original image to be tested; obtaining a first target decoding vector that matches the target feature map based on a positive sample original image, wherein the positive sample original image is an image without abnormal regions; decoding the first target decoding vector to obtain a generated image to be tested of the original image to be tested. Through the technical solution provided by the embodiments of the present disclosure, the decoding vector to be decoded of the original image to be tested is obtained through a positive sample original image without abnormal regions, so that the original image to be tested can be reconstructed more accurately, avoiding reconstruction of abnormal regions of the original image to be tested.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Digital X-ray imaging technology (abbreviated as DR (digital radiography) imaging) has a very wide range of applications in the fields of medical treatment and security inspection. In the customs inspection of vehicles at land and sea ports, DR imaging can clearly present the internal structure of the vehicle, thus becoming one of the main auxiliary inspection technologies. Inspectors can identify abnormal items in the vehicle by observing DR images, but this manual inspection method will cause problems such as inefficiency and waste of resources.

[0003] Detecting abnormal entrainment in vehicles can be achieved through traditional subtraction technology. The subtraction method uses the empty vehicle registration map of the same model as a reference, subtracting the empty vehicle image from the image to be tested, thereby highlighting the abnormal area. However, due to the variety of vehicle models and deviations in scanning angles during actual security inspections, it is difficult to obtain a completely corresponding empty vehicle registration map, and the effect of the registration map method is greatly limited.

[0004] In addition, with the comprehensive development of deep learning technology, the recognition of abnormal entrainment in images can be completed through trained neural network models. However, the trained neural network model in the related art sometimes reconstructs the abnormal area of ​​the original input image better. In this way, when the trained neural network model is applied to the application scenario of abnormal area detection (such as the above-mentioned recognition of abnormal entrainment in the vehicle DR image), it may cause the problem of missing the abnormal area in the original input image.

[0005] Therefore, a new image processing method, image processing device, computer-readable storage medium and electronic device are needed.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The purpose of the embodiments of the present disclosure is to provide an image processing method, device, computer-readable storage medium and electronic device, thereby overcoming the technical problem existing in the related art of possible reconstruction of abnormal areas of the original image to be measured, at least to a certain extent.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0009] An embodiment of the present disclosure provides an image processing method, which includes: acquiring an original image to be tested; encoding the original image to be tested to generate a target feature map of the original image to be tested; obtaining a first target decoding vector that matches the target feature map based on a positive sample original image, wherein the positive sample original image is an image without abnormal areas; decoding the first target decoding vector to obtain a generated image to be tested of the original image to be tested.

[0010] The disclosed embodiment provides an image processing device, which includes: an original image acquisition unit to be tested, used to acquire the original image to be tested; a target feature map generation unit, used to encode the original image to be tested and generate a target feature map of the original image to be tested; a first target decoding vector acquisition unit, used to obtain a first target decoding vector matching the target feature map based on a positive sample original image, wherein the positive sample original image is an image without abnormal areas; and a generated image to be tested acquisition unit, used to decode the first target decoding vector to obtain a generated image to be tested of the original image to be tested.

[0011] An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method described in the above embodiment is implemented.

[0012] An embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the above embodiments.

[0013] In the technical solutions provided by some embodiments of the present disclosure, a first target decoding vector matching the target feature map of the original image to be tested is obtained through a positive sample original image. Since the first target decoding vector of the original image to be tested is obtained from the positive sample original image and no longer contains the abnormal region in the original image to be tested, the generated image to be tested generated after decoding the first target decoding vector can avoid reconstruction of the abnormal region in the original image to be tested. When the technical solutions provided by the embodiments of the present disclosure are applied to application scenarios where abnormal region detection is required, the detection accuracy of abnormal regions in the original image to be tested can be improved.

[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0016] Figure 1 A schematic diagram showing an exemplary system architecture to which the image processing method or image processing device according to an embodiment of the present disclosure can be applied;

[0017] Figure 2 A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown;

[0018] Figure 3 The flowchart of the image processing method according to one embodiment of the present disclosure is schematically shown;

[0019] Figure 4 The flowchart of the image processing method according to one embodiment of the present disclosure is schematically shown;

[0020] Figure 5 A schematic diagram schematically shows an image processing method according to an embodiment of the present disclosure applied to abnormality detection;

[0021] Figure 6 The flowchart of the image processing method according to one embodiment of the present disclosure is schematically shown;

[0022] Figure 7 A schematic diagram of a model structure of an autoencoder according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 8 A schematic diagram of a network for local feature coding according to an embodiment of the present disclosure is shown;

[0024] Fig. 9 A schematic diagram schematically shows a storage space search according to an embodiment of the present disclosure;

[0025] Fig.10 The block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0027] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0030] Figure 1 The diagram shows an exemplary system architecture 100 to which the image processing method or image processing apparatus according to the embodiment of the present disclosure can be applied.

[0031] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. For example, the server 105 may be a server cluster composed of multiple servers, a cloud server, etc. The server 105 may be a server that provides various services.

[0033] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, wearable smart devices, smart home devices, desktop computers, etc.

[0034] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0035] It should be noted that Figure 2 The computer system 200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0036] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. Various programs and data required for system operation are also stored in the RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0037] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0038] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 209, and / or installed from a removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the system of the present application are executed.

[0039] It should be noted that the computer-readable storage medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0040] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0041] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0042] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment. For example, the electronic device may implement the following Figure 3 or Figure 4 or Figure 6 The steps shown.

[0043] In one embodiment of the present disclosure, an image processing method is proposed to optimize the technical problems existing in the above-mentioned related technologies. Figure 3 As shown, the image processing method is applicable to the electronic device described in the above embodiment, and can be Figure 1 The process may be executed by the terminal device 101 (or the terminal devices 102 and 103) or the server 105, or may be executed interactively between the terminal device 101 and the server 105, which is not limited in the present disclosure. Figure 3 The method provided in the embodiment may include the following steps S310 to S340:

[0044] In step S310, an original image to be tested is obtained.

[0045] In the embodiments of the present disclosure, the original image to be tested is taken as the DR image of the current vehicle to be tested. The current vehicle to be tested is assumed to be the vehicle to be identified as having abnormal entrainment. The specific situation of abnormal entrainment can be determined according to different scenarios such as medical treatment and security inspection. However, the present disclosure is not limited to this. When the technical solutions provided in the embodiments of the present disclosure are applied to different scenarios, the original image to be tested can be any one or more types of images of the current object to be tested.

[0046] In the disclosed embodiment, data enhancement (e.g., histogram normalization) may be performed on the acquired original image to be tested, and the size of the original image to be tested, such as width and height, may be adjusted to a uniform size, such as W×H, where W is a uniform width, H is a uniform height, and both W and H are real numbers greater than 0.

[0047] In step S320, the original image to be tested is encoded to generate a target feature map of the original image to be tested.

[0048] In the embodiments of the present disclosure, any supervised, semi-supervised or unsupervised neural network model can be used to encode the original image to be tested. Considering that supervised neural network models such as target detection and segmentation need to rely on a large amount of manual image annotation, and there are complex types of foreign objects and no fixed form, it is even more difficult to identify foreign objects that have not appeared in the training process. In the following embodiments, the encoder in the trained autoencoder (Autoencoder) is used as an example to encode the original image to be tested, but the present disclosure is not limited to this. The technical solution provided in the embodiments of the present disclosure can use any type and structure of suitable encoders and decoders.

[0049] Among them, the autoencoder is an unsupervised neural network model that can be used in the field of unsupervised anomaly detection. In the unsupervised case, there is usually a lack of abnormal samples (sample images with abnormal areas, also known as negative sample original images) to learn. The basic assumption of the algorithm is that normal samples (sample images without abnormal areas, also known as positive sample original images) and abnormal samples follow different distributions. The autoencoder trained according to the positive sample original image can better reconstruct and restore the normal area in the original image to be tested, but it cannot better reconstruct and restore the abnormal area in the original image to be tested. If there is an abnormal area in the original image to be tested, the reconstruction error between the generated image to be tested reconstructed by the trained autoencoder and the original image to be tested is large, so that the abnormal area in the original image to be tested can be automatically identified based on the large reconstruction error.

[0050] In an exemplary embodiment, encoding the original image to be tested to generate a target feature map of the original image to be tested may include: processing the original image to be tested by an encoder of an autoencoder to obtain a fused feature map of the original image to be tested; and downsampling the fused feature map of the original image to be tested to obtain the target feature map.

[0051] In an exemplary embodiment, the encoder may include a convolutional encoding network, which may include a first convolution unit, a second convolution unit, and a third convolution unit.

[0052] In the disclosed embodiment, the convolutional coding network is an extension of the autoencoder, which uses the excellent performance of the convolutional network in extracting and representing image features to replace the fully connected network of the traditional autoencoder. In the disclosed embodiment, a convolutional coding network including three convolutional units (referred to as the first convolutional unit, the second convolutional unit, and the third convolutional unit) is used as an example for illustration, but the disclosure is not limited thereto, and convolutional units of corresponding depths can be set according to actual needs.

[0053] In an exemplary embodiment, the original image to be tested is processed by the encoder of the autoencoder to obtain a fused feature map of the original image to be tested, which may include: processing the original image to be tested by the first convolution unit to obtain a first feature map of the original image to be tested; processing the first feature map by the second convolution unit to obtain a second feature map of the original image to be tested; processing the second feature map by the third convolution unit to obtain a third feature map of the original image to be tested; aggregating the first feature map, the second feature map and the third feature map of the original image to be tested to obtain a fused feature map of the original image to be tested. The disclosed embodiment may fuse information of the feature maps output by the three convolution units in the encoder of the autoencoder (referred to as the first feature map, the second feature map and the third feature map, respectively) to obtain a fused feature map of the input original image to be tested. For details, please refer to the following Figure 7 The encoder structure diagram is shown.

[0054] In an exemplary embodiment, the first convolution unit, the second convolution unit and the third convolution unit may be residually connected.

[0055] To a certain extent, the deeper the depth of the neural network model, the stronger the expression ability and the better the performance. However, as the network deepens, during the back propagation process, gradient multiplication is prone to cause the problem of gradient vanishing, making the network unable to be effectively trained. Therefore, in the embodiment of the present disclosure, a residual connection is introduced in the encoder, that is, a skip connection (shortcut) is superimposed on the convolution layer in the ordinary convolution unit, then the corresponding convolution layer adds a constant term when calculating the partial derivative, and during the back propagation process, gradient multiplication will not cause the gradient to vanish.

[0056] In an exemplary embodiment, downsampling the fused feature map of the original image to be tested to obtain the target feature map of the original image to be tested may include: downsampling the fused feature map of the original image to be tested using a first sub-pixel convolutional layer to obtain the target feature map of the original image to be tested.

[0057] The encoding process requires downsampling. Traditional methods use convolution or Pooling operations for downsampling, which is often accompanied by certain information loss. In order to improve the network effect, the embodiment of the present disclosure introduces a first sub-pixel convolution layer after the encoder for downsampling, and uses periodic sampling to sample the spatial dimension to the channel dimension to obtain the target feature map after downsampling.

[0058] In step S330, a first target decoding vector matching the target feature map is obtained according to the positive sample original image, wherein the positive sample original image is an image without abnormal regions.

[0059] In the embodiment of the present disclosure, the identification of abnormal areas in DR images is used as an example for illustration. In this case, the abnormal area may refer to an area in the DR image of the current vehicle to be tested where abnormal entrainment exists, which may also be referred to as a foreign body area, and an area where there is no abnormal entrainment is referred to as a normal area. However, the present disclosure is not limited thereto, and the division of normal areas and abnormal areas in the image is set according to actual application scenarios.

[0060] In the embodiment of the present disclosure, a storage space can be pre-divided (in the following embodiments, memory space is used as an example for illustration, but the present disclosure is not limited thereto, and it can be any storage space, which can be a local storage space or a non-local storage space) for storing decoding vectors, which is called decoding vector storage space. In the process of training the autoencoder, after the positive sample original image is processed by the encoder of the autoencoder, the sample feature vector of the positive sample original image can be obtained. After the training is completed, the clustering center of all sample feature vectors of each positive sample original image can be automatically learned. After the target feature map of the original image to be tested is obtained by the encoder of the autoencoder, the clustering center of the sample feature vector of the positive sample original image that matches it (for example, the highest similarity or the similarity exceeds the set threshold, the calculation of the similarity between vectors can adopt any one or a combination of multiple methods such as Euclidean distance and Hamming distance, which is not limited by the present disclosure) can be searched from the above decoding vector storage space as the target clustering center to generate the first target decoding vector.

[0061] In an exemplary embodiment, obtaining a first target decoding vector that matches the target feature map based on a positive sample original image may include: determining a target decoding vector storage subspace of the original image to be tested from a decoding vector storage space, the target decoding vector storage subspace including a decoding vector storage module, wherein the decoding vector storage module is used to store cluster centers of sample feature vectors in a space corresponding to the positive sample original image; spatially dividing the target feature map to obtain target subregions of the target feature map; encoding each target subregion to obtain a target feature vector of each target subregion; respectively retrieving the decoding vector storage modules of the corresponding space according to the target feature vector of each target subregion to obtain the target cluster centers of each target subregion; and aggregating the target cluster centers of each target subregion to generate the first target decoding vector.

[0062] Considering that the autoencoder in the related art has a good effect on low-resolution image processing, but has limited ability to process high-resolution images. The DR image acquisition resolution of the vehicle often reaches thousands of pixels. If the autoencoder in the related art is used to reconstruct the entire image, the effect will not meet the standard, and the reconstruction will lose image details after image downsampling. Therefore, in the embodiment of the present disclosure, the autoencoder in the related art is improved to a certain extent, and the target feature map of the input high-resolution original image to be tested is spatially divided into multiple target sub-areas. Correspondingly, in the training process, the sample feature vectors of the high-resolution positive sample original image can be similarly spatially divided, and the clustering centers of the sample feature vectors of the corresponding space of each positive sample original image are stored in the decoding vector storage module of the corresponding space. In this way, when retrieving the target clustering center of the target sub-area of ​​the corresponding space of the original image to be tested, it can be searched in the decoding vector storage module of the corresponding space. That is, on the basis of the input high-resolution original image (which can include the original image to be tested and the positive sample original image), encoding is performed based on local features, and local feature encoding is introduced to enrich the encoding space information, thereby ensuring the reconstruction quality of the high-resolution original image to be tested and the positive sample original image. For details, please refer to the following Figure 8 Description.

[0063] In an exemplary embodiment, determining a target decoding vector storage subspace of the original image to be tested from a decoding vector storage space may include: determining a target vehicle model corresponding to the original image to be tested; determining a target decoding vector storage subspace corresponding to the target vehicle model from the decoding vector storage space, wherein the target decoding vector storage subspace stores cluster centers of sample feature vectors of positive sample original images corresponding to the target vehicle model.

[0064] Considering that in actual situations, there are many types of vehicles in top-view vehicle DR images, and the images of different vehicle models are very different, sharing the same decoding vector storage space will interfere with each other. In the embodiment of the present disclosure, during the training process, the positive sample original images can be classified into vehicle models to determine their vehicle model information, and the decoding vector storage space can be isolated based on different vehicle model information. For example, if all the positive sample original images include a certain number of vehicle models (assuming that there are N+1 types, where N is an integer greater than or equal to 0), the decoding vector storage space is divided into N+1 decoding vector storage subspaces, for example, Fig. 9As shown, assuming that cls0 represents the decoding vector storage subspace corresponding to the first type of vehicle model, which is used to store the cluster center of the sample feature vector of the positive sample original image corresponding to the first type of vehicle model; cls1 represents the decoding vector storage subspace corresponding to the second type of vehicle model, which is used to store the cluster center of the sample feature vector of the positive sample original image corresponding to the second type of vehicle model; ..., clsN represents the decoding vector storage subspace corresponding to the N+1 type of vehicle model, which is used to store the cluster center of the sample feature vector of the positive sample original image corresponding to the N+1 type of vehicle model. The positive sample original images of each type of vehicle model have their own storage space, such as memory space.

[0065] When the target cluster center of the original image to be tested is to be retrieved from the decoding vector storage space, the target vehicle model of the original image to be tested can be determined first, and then the target decoding vector storage subspace can be determined according to the target vehicle model. Fig. 9 In the embodiment, the target decoding vector storage subspace is taken as cls1 as an example for illustration, that is, it is assumed that the target vehicle model corresponding to the original image to be tested is the second vehicle model.

[0066] In step S340, the first target decoding vector is decoded to obtain the generated image to be tested of the original image to be tested.

[0067] In an exemplary embodiment, decoding the first target decoding vector to obtain the generated image to be tested of the original image to be tested may include: upsampling the first target decoding vector to obtain an upsampled feature map of the original image to be tested; processing the upsampled feature map of the original image to be tested by a decoder of an autoencoder to obtain the generated image to be tested of the original image to be tested.

[0068] In an exemplary embodiment, upsampling the first target decoding vector to obtain an upsampled feature map of the original image to be tested may include: upsampling the first target decoding vector using a second sub-pixel convolutional layer to obtain an upsampled feature map of the original image to be tested.

[0069] In an exemplary embodiment, the decoder may include a convolutional decoding network.

[0070] In an exemplary embodiment, the convolutional units in the convolutional decoding network may be residually connected.

[0071] In the disclosed embodiment, the process of convolution decoding performed by the convolution decoding network is the reverse process of convolution encoding of the convolution encoding network. The convolution units in the convolution decoding network may also adopt residual connections, for example, Figure 7 As shown, a convolutional decoding network also includes three convolutional units for illustration, but the present disclosure is not limited to this.

[0072] Unlike the convolutional coding network, the convolutional decoding network needs to be upsampled. In the related art, deconvolution is used for upsampling. First, the resolution of the feature map needs to be improved, which increases the amount of calculation. Secondly, interpolation is required to convert the low-resolution space into the high-resolution space. Since each layer of information comes from the previous layer, it will not bring additional information that is conducive to reconstruction. In the embodiment of the present disclosure, in order to improve the network effect, a second sub-pixel convolution layer is introduced.

[0073] In the embodiment of the present disclosure, let the upsampling factor be c, where c is a real number greater than 0. First, ordinary convolution is performed to obtain c. 2 The feature map in the channel direction is then sampled into the spatial dimension using periodic sampling to obtain a high-resolution image. In this way, end-to-end learning can be performed as long as the target loss function is defined.

[0074] The image processing method provided by the embodiment of the present disclosure obtains a first target decoding vector that matches the target feature map of the original image to be tested through the positive sample original image. Since the first target decoding vector of the original image to be tested is obtained from the positive sample original image and no longer contains the abnormal area in the original image to be tested, the generated image to be tested generated after decoding the first target decoding vector can avoid the reconstruction of the abnormal area in the original image to be tested. When the technical solution provided by the embodiment of the present disclosure is applied to an application scenario where abnormal area detection is required, the detection accuracy of the abnormal area in the original image to be tested can be improved.

[0075] Figure 4 Embodiment The above Figure 3 The technical solution proposed in the embodiment is applied to the detection of abnormal areas in the original image to be tested. Figure 4 As shown, the execution of steps S310-340 can refer to the above Figure 3 Embodiment, with the above Figure 3 Compared with the embodiment, the difference is that it can further include the following steps.

[0076] In step S410, a difference image is generated according to the original image to be tested and the generated image to be tested.

[0077] For example, Figure 5 As shown, after the original image to be tested is processed by the encoder and the decoder, the generated image to be tested is reconstructed, and the original image to be tested is subtracted from the generated image to be tested to obtain a difference image.

[0078] In step S420, an abnormal area in the original image to be tested is identified according to the difference image.

[0079] For example, Figure 5Based on the difference image obtained above, the reconstruction or reconstruction error between the original image to be tested and the generated image to be tested can be located. If there is an abnormal area with abnormal entrainment in the original image to be tested, since the first target decoding vector is obtained based on the positive sample original image, the abnormal area in the original image to be tested is eliminated, and the generated image to be tested is better reconstructed from the normal area in the original image to be tested. Therefore, the reconstruction error between the original image to be tested and the generated image to be tested is large. Therefore, the abnormal area in the original image to be tested can be determined based on the reconstruction error.

[0080] Figure 6 The flowchart of the image processing method according to an embodiment of the present disclosure is schematically shown. Figure 6 As shown above Figure 3 Compared with the embodiment, the difference is that it can further include the following steps.

[0081] In step S610, the positive sample original image is obtained.

[0082] In an exemplary embodiment, obtaining the positive sample original image may include: obtaining digital X-ray images (DR images) of various vehicle types in an empty vehicle state; determining a standardized reference image from the digital X-ray images; and using the standardized reference image to perform histogram standardization processing on the digital X-ray images to obtain the positive sample original image.

[0083] In the disclosed embodiment, the DR image of the vehicle is used as an example to illustrate that DR images of different vehicle models in an empty vehicle state are collected. In order to enable the trained autoencoder to recognize various vehicle models, as many vehicle models as possible are included in the collection. Each vehicle model is collected several times to avoid the influence of the instability of the collection equipment as much as possible. Alternatively, a normal DR image without entrainment can be directly obtained from the vehicle DR imaging database as the positive sample original image.

[0084] In the disclosed embodiment, the sizes of all positive sample original images may also be adjusted to a uniform size such as W×H.

[0085] In the disclosed embodiment, data enhancement can also be performed on the above-collected DR images. In actual situations, the grayscale variation range of the DR images collected by various field collection devices is different. Some field-collected DR images are generally very dark and difficult to identify details. Histogram regulation can be used to keep the pixel distribution of the collected DR images roughly consistent.

[0086] For example, from all positive sample original images with a width and height size of W×H, an image with imaging pixel values ​​within a reasonable range can be statistically selected as a normalized reference image, and all positive sample original images can be normalized using histogram normalization.

[0087] In step S620, the target loss function of the autoencoder is determined.

[0088] In an exemplary embodiment, the target loss function may include perceptual loss. Determining the target loss function of the autoencoder may include: processing the positive sample original image through a trained pre-trained model, extracting the output features of the positive sample original image from the jth layer of the pre-trained model, where j is a positive integer greater than or equal to 1; processing the positive sample original image through the autoencoder to obtain a positive sample generated image of the positive sample original image; processing the positive sample generated image through the trained pre-trained model, extracting the output features of the positive sample generated image from the jth layer of the pre-trained model; determining the perceptual loss based on the output features of the positive sample original image and the output features of the positive sample generated image.

[0089] In an exemplary embodiment, the target loss function includes a coding loss. Determining the target loss function of the autoencoder may include: processing the positive sample original image through the encoder of the autoencoder to obtain a coding vector of the positive sample original image; determining a second target decoding vector matching the coding vector of the positive sample original image from the decoding vector storage space; obtaining a first termination error back propagation parameter of the second target decoding vector; and determining the coding loss according to the coding vector of the positive sample original image and the first termination error back propagation parameter.

[0090] In an exemplary embodiment, the target loss function includes a decoding vector loss. Determining the target loss function of the autoencoder may include: processing the positive sample original image through the encoder of the autoencoder to obtain the encoding vector of the positive sample original image; obtaining a second termination error backpropagation parameter of the encoding vector of the positive sample original image; determining a second target decoding vector matching the encoding vector of the positive sample original image from the decoding vector storage space; and determining the decoding vector loss according to the second termination error backpropagation parameter and the second target decoding vector.

[0091] In the embodiment of the present disclosure, the trainable parameters of the network may include three parts: Encoder, Decoder and Decoding Vector Module (which may be expressed as Embedding Module). Correspondingly, the target loss function Loss all It can include the following 3 parts:

[0092] Loss all =Loss PCT +LossEncoder +Loss embedding (1)

[0093] Among them, in the above formula (1), Loss PCT Represents the reconstruction error of the decoder, which is represented by perceptual loss. Encoder Represents the encoding loss of the encoder, Loss embedding Denotes the decoding vector loss. They are described below.

[0094] 1) Perceptual Loss PCT ):

[0095] In the field of image reconstruction, most related technologies use Mean Squared Error (MSE), but due to the design problem of MSE itself, the details of the image generated by the positive sample are blurred. In the disclosed embodiment, perceptual loss (PCT-Loss) is used to represent the image reconstruction error. Perceptual loss does not depend on the difference between the pixels of the image, but depends on the difference between the high-level features of the image, which can well reconstruct the details of the original image of the positive sample.

[0096] In the embodiment of the present disclosure, during the training process of the autoencoder, a pre-trained model is introduced, and the parameters of the pre-trained model are fixed or frozen, and do not participate in the training of the autoencoder. Assume that the pre-trained model uses the function Represented by, then the j-th layer of the pre-trained model is represented by j is a positive integer greater than or equal to 1, then the perceptual loss of the jth layer of the pre-trained model is As shown in the following formula (2):

[0097]

[0098] In the above formula (2), y and They represent the positive sample original image and the positive sample generated image output after the autoencoder processes the positive sample original image; C j , W j , H j They respectively represent the number of channels, width, and height of the feature map output by the j-th layer of the pre-trained model.

[0099] For example, Figure 7 As shown, it is assumed that a pre-trained VGG (Visual Geometry Group Network) model is used as a pre-trained model of an embodiment of the present disclosure. For example, a standard VGG19 model can be selected, but the present disclosure is not limited to this.

[0100] In the process of training the autoencoder, the parameters of the pre-trained VGG model are fixed, the positive sample original image y is input into the pre-trained VGG model, the Conv3 layer (the third convolution layer or convolution layer 3, i.e., assuming j=3, but the present disclosure is not limited to this) of the pre-trained VGG model is taken as the output, and the output features of the positive sample original image y are obtained; then the positive sample generated image output by the decoder is Input to the pre-trained VGG model, take the Conv3 layer of the pre-trained VGG model as the output, and obtain the positive sample generated image The output features of the image are then used to calculate the perceptual loss according to the above formula (2). That is, the VGG model is selected as the perceptual loss feature extractor, the output of the convolutional layer 3 of the VGG model is fixed, the output features of the positive sample original image and the positive sample generated image are extracted respectively, and the L2 norm between the two output features is used as the perceptual loss. The encoding loss and decoding vector loss described below are combined to train the autoencoder.

[0101] 2) Coding Loss Encoder ):

[0102] In the embodiment of the present disclosure, the coding loss can be calculated according to the following formula:

[0103]

[0104] In the above formula (3), z(y) is the encoding vector generated after the positive sample original image is processed by the encoder of the autoencoder; e is the second target decoding vector matched with the encoding vector of the positive sample original image searched from the decoding vector storage space; β is a preset parameter, for example, it can be set to 0.25, but the present disclosure is not limited thereto; SG(e) represents the first termination error back propagation parameter. The parameters of the Encoder part can be trained according to the above encoding loss.

[0105] 3) Decoding vector loss (Loss embedding ):

[0106] In the embodiment of the present disclosure, the decoding vector loss can be calculated according to the following formula:

[0107]

[0108] In the above formula (4), SG(z(y)) represents the second termination error back propagation parameter of the encoding vector of the positive sample original image. The parameters of the decoding vector storage space can be trained according to the decoding vector loss to determine the clustering center of the sample feature vector of the positive sample original image stored therein.

[0109] In step S630, the autoencoder is trained according to the positive sample original image and the target loss function to determine the cluster center of the sample feature vector of the positive sample original image stored in the decoding vector storage space in the autoencoder.

[0110] In the disclosed embodiment, the DR image data of a normal unentrained vehicle can be used to generate a positive sample original image, and a pre-trained model can be introduced to define a target loss function and train the autoencoder end-to-end. The parameter optimization method can include any one of the various methods such as the stochastic gradient descent method, the momentum method, and the Adam method (a first-order optimization algorithm that can replace the stochastic gradient descent process).

[0111] For the autoencoder in the related art, if the abnormal sample and the normal sample have the same composition pattern, such as some local edges; or the trained decoder has too strong decoding ability for the abnormal sample, the abnormal area in the abnormal sample can also be well reconstructed. In order to overcome this shortcoming, a decoding vector storage space is proposed in the embodiment of the present disclosure to enhance the ability of the autoencoder. The decoding vector storage space after training according to the target loss function tends to obtain the cluster center of the sample feature vector of all positive sample original images. For a given input, the encoding vector obtained by the encoder is not directly input into the decoder, but it is retrieved in the decoding vector storage space to find the most relevant target cluster center, and then these target cluster centers are aggregated and input into the decoder for decoding.

[0112] In an exemplary embodiment, training the autoencoder according to the positive sample original image and the target loss function, and determining the clustering center of the sample feature vector of the positive sample original image stored in the decoding vector storage space in the autoencoder may include: determining the vehicle model information of the positive sample original image; inputting the positive sample original image and its vehicle model information into the autoencoder, and training the autoencoder according to the target loss function, wherein the decoding vector storage space is divided into independent decoding vector storage subspaces corresponding to each vehicle model information according to the vehicle model information of the positive sample original image, and the clustering center of the sample feature vector of the positive sample original image corresponding to each vehicle model information is stored in the decoding vector storage subspace.

[0113] In the embodiment of the present disclosure, a trained vehicle type classification model can be used (for example, a VGG model can also be selected, but the present disclosure is not limited to this, and it can be any model that can achieve vehicle type classification) to determine the vehicle type information of each positive sample original image, and then when the autoencoder is trained, the positive sample original image and its vehicle type information are input into the autoencoder for end-to-end training, and the decoding vector storage space is divided and isolated into multiple decoding vector storage subspaces according to the vehicle type information.

[0114] In the disclosed embodiment, a training data set of a vehicle model classification model can be obtained, the training data set includes training images and vehicle model label information thereof, each training image is input into the vehicle model classification model, the vehicle model classification model predicts and outputs the predicted vehicle model information corresponding to the training image, and the loss function of the vehicle model classification model is obtained according to the vehicle model label information of the training image and its predicted vehicle model information, and the model parameters of the vehicle model classification model are optimized to minimize the loss function of the vehicle model classification model, thereby obtaining a trained vehicle model classification model. The trained vehicle model classification model can also be used to predict the target vehicle model of the original image to be tested.

[0115] In an exemplary embodiment, the decoding vector storage space may include a decoding vector storage module, wherein the decoding vector storage module is used to store the cluster center of the sample feature vector in the space corresponding to the positive sample original image.

[0116] In the decoding vector storage space, if the sample feature vectors of all positive sample original images share the same decoding vector storage space, that is, share the cluster center, interference will occur between the positive sample original images of different vehicle types and the sample feature vectors of different spatial positions on the same positive sample original image. Therefore, when modeling the decoding vector storage space, the embodiment of the present disclosure can isolate the decoding vector storage space according to the vehicle type information and the spatial position. For example, the decoding vector storage space can be divided into at least one decoding vector storage subspace according to the vehicle type information, and each decoding vector storage subspace can be further divided into at least one decoding vector storage module according to the division of spatial position. During the training and testing of the autoencoder, each decoding vector storage subspace is independent of each other.

[0117] For example, if the size of the decoding vector storage space is assumed to be 100×64, and 10 types of vehicle models are predefined, the 100×64 decoding vector storage space is divided into 10 10×64 decoding vector storage subspaces, the first 10×64 decoding vector storage subspace corresponds to the first type of vehicle model, ..., the tenth 10×64 decoding vector storage subspace corresponds to the tenth type of vehicle model. Assuming that 1000 positive sample original images are trained and 100 64×1 cluster centers are obtained, the 100 64×1 cluster centers are stored in the storage vector storage subspaces of the corresponding vehicle models. If the target vehicle model of the original image to be tested is the first type of vehicle model, the target cluster center is searched in the first 10×64 decoding vector storage subspace.

[0118] The execution of steps S310-340 can refer to the above Figure 3 Example.

[0119] In the disclosed embodiment, after the autoencoder training is completed, the network parameters of the autoencoder will be fixed, the pre-trained model used to calculate the perceptual loss will be removed, and the Encoder, Decoder and Embedding Module will be retained. The original image to be tested is input into the trained autoencoder to obtain the generated image to be tested. The difference image between the generated image to be tested and the original image to be tested is compared, and the position of the abnormal area in the original image to be tested is determined according to the size of the reconstructed difference.

[0120] The image processing method proposed in the embodiment of the present disclosure can be an unsupervised abnormality detection method for vehicle DR images based on an autoencoder. In the training stage, the autoencoder is trained using the acquired normal positive sample original image without entrainment; in the test stage or the online use stage, the normal area in the original image to be tested can be well restored by the autoencoder, while the abnormal area in the original image to be tested cannot be well restored, so that the abnormal area in the original image to be tested can be located based on the size of the reconstruction error between the original image to be tested and the generated image to be tested. The method provided in the embodiment of the present disclosure does not require a large amount of manual annotation, and can theoretically identify all types of entrainment types.

[0121] Figure 8 A network diagram of local feature coding according to an embodiment of the present disclosure is schematically shown.

[0122] like Figure 8 As shown, in the embodiment of the present disclosure, the target feature map is divided into 9 target sub-areas as an example for illustration, and correspondingly, each decoding vector storage subspace includes 9 decoding vector storage modules. It can be understood that the 9 target sub-areas here are only used for illustration, and the present disclosure is not limited thereto, and can be set according to actual needs.

[0123] The encoder is a process of multiple downsampling, and the resolution of the feature map (for example, the first feature map to the third feature map) output by each convolution unit gradually decreases, and is finally encoded into a target feature map. The size of the original image to be tested and the original image of the positive sample processed in the embodiment of the present disclosure is large and the texture is complex. Too many downsampling times are not easy to restore the detailed information of the input image, and too few downsampling times will cause the final target feature map or the fully connected layer parameter of the encoding vector to be too large. Therefore, in the embodiment of the present disclosure, unlike the standard autoencoder, an autoencoder based on local feature encoding is proposed. After the input original image to be tested passes through the convolutional coding network, a target feature map with a downsampling rate of 1 / 8 is formed (as described above, the first sub-pixel convolution layer can also be further used for downsampling), and the target feature map is evenly divided into 9 target sub-regions of 8×8 size, and each target sub-region corresponds to a decoding vector storage module of a decoding vector storage space, which realizes the correlation between the decoding vector storage space and the spatial position of each target sub-region. Then, each target sub-region can be encoded in parallel to obtain the feature maps of 9 target sub-regions respectively. The feature maps of the nine 8×8 target sub-regions are flattened to generate target feature vectors of fixed length, for example, nine 64×1 target feature vectors. These nine 64×1 target feature vectors are then input into the decoding vector storage space, and the target cluster center of the target sub-region of the corresponding space is searched in the decoding vector storage module of the corresponding space.

[0124] The following is an example of applying the image processing method proposed in the above embodiment to the DR image of the current vehicle to be tested as the original image to be tested to perform abnormality detection:

[0125] First, obtain the training data set of the autoencoder. Here, it is assumed that 8133 normal images without entrainment are obtained from the vehicle DR imaging database as the positive sample original images, and the width and height of each image are adjusted to 512×256 (that is, assuming W=512, H=256). From them, an image with a reasonable imaging pixel value is statistically selected as the normalized reference image, and all images are normalized by applying histogram normalization.

[0126] A VGG model is trained as a vehicle type classification model, and 8133 positive sample original images are classified into vehicle types. Here, it is assumed that a total of 368 vehicle types are obtained through statistics. The positive sample original images and their vehicle type information are input into the autoencoder for end-to-end training.

[0127] 2. Network structure:

[0128] Specifically, a 512×256 positive sample original image is input to the convolutional coding network, and the convolutional units and residual connections of the convolutional coding network are used for feature extraction. The first sub-pixel convolutional layer performs three downsamplings to finally obtain a target feature map of 32×64×64. A batch normalization operation can be performed after each convolutional unit of the convolutional coding network to accelerate network convergence and improve network stability.

[0129] The target feature map of size 32×64×64 is spatially divided. Assuming that the target feature map of size 32×64×64 obtained by the convolutional coding network is divided in space based on a window of size 8×8, 32 target sub-regions of size 8×8×64 are obtained.

[0130] Assume that the decoding vector storage module size is defined as 100×64, the number of cluster centers is 100, and the cluster center length is 64. Based on the spatial information (32 spatial positions) and vehicle model information (368 vehicle models), the decoding vector storage space is modeled to obtain a decoding vector storage space of 368×32×100×64.

[0131] During the training and testing process, the parameter updates in each decoding vector storage subspace do not affect each other. The most relevant target cluster center is searched for the 32 target sub-areas in the target vehicle model's target decoding vector storage subspace, and then aggregated to obtain a 32×64×64 target decoding vector (the second target decoding vector in the training process or the first target decoding vector in the test process) and input into the convolutional decoding network.

[0132] Since the convolutional decoding network and the convolutional encoding network are mirror images of each other, a 512×256 generated image to be tested or a positive sample generated image can be obtained in the end.

[0133] Here we assume that the standard VGG19 model is selected as the pre-trained model and used as a perceptual loss feature extractor. The Conv3 layer output of the standard VGG19 model is fixed to extract the output features of the original image and the output features of the generated image respectively, and the L2 norm between the output features of the original image and the output features of the generated image is used as the perceptual loss. The encoding loss and decoding vector loss are combined to train the autoencoder.

[0134] The training is set here to use the RMS optimizer (an effective and commonly used deep neural network optimization algorithm), the learning rate is set to 0.003, the batch size is set to 3, and a maximum of 100 epochs are trained (one epoch is when all the positive sample original images in the training data set are trained). The target loss function curve and the positive sample generated image effect are simultaneously observed in TensorBoard (a powerful visualization tool that comes with Tensorflow and is also a web application suite). When the curve drops to a stable change stage, it can be terminated in advance to prevent network overfitting.

[0135] In the testing phase, the branch of perceptual loss, i.e., the pre-trained model, is removed, the model parameters of the autoencoder are fixed, 392 original images to be tested are tested, the generated image to be tested and the original image to be tested are subtracted to generate a difference image, a reasonable threshold is set, the area of ​​the difference image that exceeds the threshold is calculated, and certain post-processing is performed to identify suspected abnormal areas.

[0136] The embodiments of the present disclosure provide an image processing method that can be applied to unsupervised anomaly detection of vehicle DR images, and has at least the following technical effects:

[0137] First, a vehicle DR image anomaly detection method is proposed, which facilitates security personnel to quickly locate abnormal areas in vehicle DR images, improve security inspection efficiency, and save labor costs.

[0138] Secondly, the unsupervised anomaly detection method based on autoencoders does not require labeling information of abnormal areas, which reduces the difficulty of collecting data sets and saves labeling costs.

[0139] Thirdly, compared with the autoencoder in the related art, the method based on local feature encoding proposed in the embodiment of the present disclosure generates images with larger size and richer details.

[0140] Fourthly, the autoencoder based on the decoding vector storage space can better suppress the reconstruction of abnormal areas. In addition, isolating the decoding vector storage space based on vehicle model information and spatial position information is beneficial to reconstructing a more accurate image.

[0141] Fifthly, compared with the generation effect of the autoencoder based on the MSE loss function, the training method based on perceptual loss proposed in the embodiment of the present disclosure generates images with higher quality.

[0142] The following describes an apparatus embodiment of the present disclosure, which can be used to execute the above-mentioned image processing method of the present disclosure. For details not disclosed in the apparatus embodiment of the present disclosure, please refer to the above-mentioned image processing method embodiment of the present disclosure.

[0143] Fig.10Schematically shows a block diagram of an image processing device according to an embodiment of the present disclosure. Fig.10 As shown, according to an embodiment of the present disclosure, an image processing device 1000 may include: a unit 1010 for acquiring an original image to be tested, a unit 1020 for generating a target feature map, a unit 1030 for acquiring a first target decoding vector, and a unit 1040 for acquiring a generated image to be tested.

[0144] Among them, the original image acquisition unit 1010 to be tested can be used to acquire the original image to be tested. The target feature map generation unit 1020 can be used to encode the original image to be tested and generate the target feature map of the original image to be tested. The first target decoding vector acquisition unit 1030 can be used to obtain a first target decoding vector matching the target feature map according to the positive sample original image, and the positive sample original image is an image without abnormal areas. The generated image to be tested acquisition unit 1040 can be used to decode the first target decoding vector to obtain the generated image to be tested of the original image to be tested.

[0145] In an exemplary embodiment, the image processing device 1000 may further include: a difference image generating unit, which can be used to generate a difference image according to the original image to be tested and the generated image to be tested; and an abnormal area identifying unit, which can be used to identify abnormal areas in the original image to be tested according to the difference image.

[0146] In an exemplary embodiment, the target feature map generation unit 1020 may include: a fused feature map obtaining unit, which can be used to process the original image to be tested through the encoder of the autoencoder to obtain the fused feature map of the original image to be tested; and a downsampling unit, which can be used to downsample the fused feature map of the original image to be tested to obtain the target feature map.

[0147] In an exemplary embodiment, the encoder may include a convolutional coding network, and the convolutional coding network may include a first convolution unit, a second convolution unit, and a third convolution unit. Wherein, the fusion feature map obtaining unit may include: a first feature map obtaining unit, which may be used to process the original image to be tested through the first convolution unit to obtain a first feature map of the original image to be tested; a second feature map obtaining unit, which may be used to process the first feature map through the second convolution unit to obtain a second feature map of the original image to be tested; a third feature map obtaining unit, which may be used to process the second feature map through the third convolution unit to obtain a third feature map of the original image to be tested; and a fusion feature map generating unit, which may be used to aggregate the first feature map, the second feature map, and the third feature map of the original image to be tested to obtain a fusion feature map of the original image to be tested.

[0148] In an exemplary embodiment, the first convolution unit, the second convolution unit and the third convolution unit may be residually connected.

[0149] In an exemplary embodiment, the downsampling unit may include: a sub-pixel convolution layer downsampling unit, which may be used to downsample the fused feature map of the original image to be tested using a first sub-pixel convolution layer to obtain a target feature map of the original image to be tested.

[0150] In an exemplary embodiment, the first target decoding vector obtaining unit 1030 may include: a target decoding vector storage subspace determining unit, which can be used to determine the target decoding vector storage subspace of the original image to be tested from the decoding vector storage space, the target decoding vector storage subspace including a decoding vector storage module, wherein the decoding vector storage module is used to store the clustering center of the sample feature vector of the corresponding space of the positive sample original image; a target sub-region division unit, which can be used to perform spatial division on the target feature map to obtain the target sub-region of the target feature map; a target feature vector obtaining unit, which can be used to encode each target sub-region to obtain the target feature vector of each target sub-region; a target clustering center obtaining unit, which can be used to retrieve the decoding vector storage module of the corresponding space according to the target feature vector of each target sub-region, and obtain the target clustering center of each target sub-region; a first target decoding vector generating unit, which can be used to aggregate the target clustering centers of each target sub-region to generate the first target decoding vector.

[0151] In an exemplary embodiment, the target decoding vector storage subspace determination unit may include: a target vehicle type determination unit, which can be used to determine the target vehicle type corresponding to the original image to be tested; a target decoding vector storage subspace judgment unit, which can be used to determine the target decoding vector storage subspace corresponding to the target vehicle type from the decoding vector storage space, and the target decoding vector storage subspace stores the cluster center of the sample feature vector of the positive sample original image corresponding to the target vehicle type.

[0152] In an exemplary embodiment, the test generated image obtaining unit 1040 may include: an upsampling unit, which can be used to upsample the first target decoding vector to obtain an upsampled feature map of the original image to be tested; and a test generated image decoding unit, which can be used to process the upsampled feature map of the original image to be tested through a decoder of an autoencoder to obtain a test generated image of the original image to be tested.

[0153] In an exemplary embodiment, the upsampling unit may include: a sub-pixel convolution layer upsampling unit, which can be used to upsample the first target decoding vector using a second sub-pixel convolution layer to obtain an upsampled feature map of the original image to be tested.

[0154] In an exemplary embodiment, the decoder may include a convolutional decoding network.

[0155] In an exemplary embodiment, the convolutional units in the convolutional decoding network may be residually connected.

[0156] In an exemplary embodiment, the image processing device 1000 may also include: a positive sample original image acquisition unit, which can be used to acquire the positive sample original image; a target loss function determination unit, which can be used to determine the target loss function of the autoencoder; and an autoencoder training unit, which can be used to train the autoencoder according to the positive sample original image and the target loss function, and determine the clustering center of the sample feature vector of the positive sample original image stored in the decoding vector storage space in the autoencoder.

[0157] In an exemplary embodiment, the positive sample original image acquisition unit may include: a normal vehicle image acquisition unit, which can be used to acquire digital X-ray images of various vehicle models in an empty vehicle state; a standardized reference image determination unit, which can be used to determine a standardized reference image from the digital X-ray image; and a data enhancement unit, which can be used to use the standardized reference image to perform histogram standardization processing on the digital X-ray image to obtain the positive sample original image.

[0158] In an exemplary embodiment, the target loss function may include a perceptual loss. The target loss function determination unit may include: a positive sample original image output feature extraction unit, which may be used to process the positive sample original image through a trained pre-trained model, and extract the output feature of the positive sample original image from the jth layer of the pre-trained model, where j is a positive integer greater than or equal to 1; a positive sample generated image acquisition unit, which may be used to process the positive sample original image through the autoencoder to obtain a positive sample generated image of the positive sample original image; a positive sample generated image output feature extraction unit, which may be used to process the positive sample generated image through the trained pre-trained model, and extract the output feature of the positive sample generated image from the jth layer of the pre-trained model; a perceptual loss determination unit, which may be used to determine the perceptual loss based on the output feature of the positive sample original image and the output feature of the positive sample generated image.

[0159] In an exemplary embodiment, the target loss function may include a coding loss. The target loss function determination unit may include: a positive sample original image coding vector acquisition unit, which may be used to process the positive sample original image through the encoder of the autoencoder to obtain the coding vector of the positive sample original image; a second target decoding vector determination unit, which may be used to determine a second target decoding vector matching the coding vector of the positive sample original image from the decoding vector storage space; a first termination error back propagation parameter acquisition unit, which may be used to obtain a first termination error back propagation parameter of the second target decoding vector; and a coding loss determination unit, which may be used to determine the coding loss according to the coding vector of the positive sample original image and the first termination error back propagation parameter.

[0160] In an exemplary embodiment, the target loss function may include a decoding vector loss. The target loss function determination unit may include: a positive sample original image encoding vector acquisition unit, which may be used to process the positive sample original image through the encoder of the autoencoder to obtain the encoding vector of the positive sample original image; a second termination error back propagation parameter acquisition unit, which may be used to obtain a second termination error back propagation parameter of the encoding vector of the positive sample original image; a second target decoding vector acquisition unit, which may be used to determine a second target decoding vector that matches the encoding vector of the positive sample original image from the decoding vector storage space; and a decoding vector loss determination unit, which may be used to determine the decoding vector loss according to the second termination error back propagation parameter and the second target decoding vector.

[0161] In an exemplary embodiment, the autoencoder training unit may include: a vehicle model information determination unit, which can be used to determine the vehicle model information of the positive sample original image; a model training unit, which can be used to input the positive sample original image and its vehicle model information into the autoencoder, and train the autoencoder according to the target loss function, wherein the decoding vector storage space is divided into independent decoding vector storage subspaces corresponding to each vehicle model information according to the vehicle model information of the positive sample original image, and the clustering center of the sample feature vector of the positive sample original image corresponding to each vehicle model information is stored in the decoding vector storage subspace.

[0162] In an exemplary embodiment, the decoding vector storage space may include a decoding vector storage module, wherein the decoding vector storage module is used to store the cluster center of the sample feature vector in the space corresponding to the positive sample original image.

[0163] It should be noted that although several units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0164] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0165] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0166] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtain the original image to be tested; Encoding the original image to be tested to generate a target feature map of the original image to be tested; Obtaining a first target decoding vector matching the target feature map according to a positive sample original image, wherein the positive sample original image is an image without abnormal regions, comprising: searching, from cluster centers of sample feature vectors of each positive sample original image, for a cluster center of sample feature vectors of a positive sample original image matching the target feature map as a target cluster center, to generate the first target decoding vector; Decoding the first target decoding vector to obtain a generated image to be tested of the original image to be tested; generating a difference image according to the original image to be tested and the generated image to be tested; The abnormal area in the original image to be tested is identified according to the difference image.

2. The image processing method according to claim 1, characterized in that: Encoding the original image to be tested to generate a target feature map of the original image to be tested includes: Processing the original image to be tested by an encoder of the autoencoder to obtain a fusion feature map of the original image to be tested; The fused feature map of the original image to be tested is downsampled to obtain the target feature map.

3. The image processing method according to claim 2, characterized in that: The encoder includes a convolutional coding network, and the convolutional coding network includes a first convolution unit, a second convolution unit, and a third convolution unit; wherein, the original image to be tested is processed by the encoder of the autoencoder to obtain a fusion feature map of the original image to be tested, including: Processing the original image to be tested by the first convolution unit to obtain a first feature map of the original image to be tested; Processing the first feature map by the second convolution unit to obtain a second feature map of the original image to be tested; Processing the second feature map by the third convolution unit to obtain a third feature map of the original image to be tested; The first feature map, the second feature map and the third feature map of the original image to be tested are aggregated to obtain a fused feature map of the original image to be tested.

4. The image processing method according to claim 3, characterized in that: The first convolution unit, the second convolution unit and the third convolution unit are residually connected.

5. The image processing method according to claim 2, characterized in that: Downsampling the fused feature map of the original image to be tested to obtain a target feature map of the original image to be tested includes: The fused feature map of the original image to be tested is downsampled by using the first sub-pixel convolution layer to obtain a target feature map of the original image to be tested.

6. The image processing method according to claim 1, characterized in that: Obtaining a first target decoding vector matching the target feature map according to the positive sample original image, including: Determine a target decoding vector storage subspace of the original image to be tested from the decoding vector storage space, wherein the target decoding vector storage subspace includes a decoding vector storage module, wherein the decoding vector storage module is used to store the cluster center of the sample feature vector of the space corresponding to the positive sample original image; Performing spatial division on the target feature map to obtain a target sub-region of the target feature map; Encode each target sub-region to obtain a target feature vector of each target sub-region; Retrieve the decoding vector storage module of the corresponding space according to the target feature vector of each target sub-region, and obtain the target cluster center of each target sub-region; The target cluster centers of the target sub-regions are aggregated to generate the first target decoding vector.

7. The image processing method according to claim 6, characterized in that: Determining a target decoding vector storage subspace of the original image to be tested from the decoding vector storage space includes: Determine the target vehicle type corresponding to the original image to be tested; A target decoding vector storage subspace corresponding to the target vehicle model is determined from the decoding vector storage space, and the target decoding vector storage subspace stores the cluster center of the sample feature vector of the positive sample original image corresponding to the target vehicle model.

8. The image processing method according to claim 1, characterized in that: Decoding the first target decoding vector to obtain a generated image to be tested of the original image to be tested includes: Upsampling the first target decoding vector to obtain an upsampled feature map of the original image to be tested; The upsampled feature map of the original image to be tested is processed by the decoder of the autoencoder to obtain a generated image to be tested of the original image to be tested.

9. The image processing method according to claim 8, characterized in that: Upsampling the first target decoding vector to obtain an upsampled feature map of the original image to be tested includes: The first target decoding vector is upsampled by using a second sub-pixel convolutional layer to obtain an upsampled feature map of the original image to be tested.

10. The image processing method according to claim 8, characterized in that: The decoder includes a convolutional decoding network.

11. The image processing method according to claim 10, characterized in that: The convolution units in the convolutional decoding network are residually connected.

12. The image processing method according to claim 1, characterized in that: Also includes: Acquire the positive sample original image; Determine the target loss function of the autoencoder; The autoencoder is trained according to the positive sample original image and the target loss function, and a cluster center of the sample feature vector of the positive sample original image stored in the decoding vector storage space in the autoencoder is determined.

13. The image processing method according to claim 12, characterized in that: Acquiring the positive sample original image includes: Obtain digital X-ray images of various vehicle types in empty state; determining a normalized reference image from the digital radiographic image; The digital X-ray image is subjected to histogram normalization processing by using the normalized reference image to obtain the positive sample original image.

14. The image processing method according to claim 12, characterized in that: The target loss function includes a perceptual loss; wherein determining the target loss function of the autoencoder includes: Processing the positive sample original image through the trained pre-trained model, extracting the output features of the positive sample original image from the j-th layer of the pre-trained model, where j is a positive integer greater than or equal to 1; Processing the positive sample original image by the autoencoder to obtain a positive sample generated image of the positive sample original image; Processing the positive sample generated image by the trained pre-trained model, and extracting output features of the positive sample generated image from the jth layer of the pre-trained model; The perceptual loss is determined according to the output features of the positive sample original image and the output features of the positive sample generated image.

15. The image processing method according to claim 12, characterized in that: The target loss function includes encoding loss; wherein determining the target loss function of the autoencoder includes: Processing the positive sample original image through the encoder of the autoencoder to obtain an encoding vector of the positive sample original image; Determine a second target decoding vector matching the encoding vector of the positive sample original image from the decoding vector storage space; Obtaining a first termination error back propagation parameter of the second target decoding vector; The coding loss is determined according to the coding vector of the positive sample original image and the first termination error back-propagation parameter.

16. The image processing method according to claim 12, characterized in that: The target loss function includes a decoding vector loss; wherein determining the target loss function of the autoencoder includes: Processing the positive sample original image through the encoder of the autoencoder to obtain an encoding vector of the positive sample original image; Obtaining a second termination error back propagation parameter of the encoding vector of the positive sample original image; Determine a second target decoding vector matching the encoding vector of the positive sample original image from the decoding vector storage space; The decoding vector loss is determined according to the second termination error back-propagation parameter and the second target decoding vector.

17. The image processing method according to claim 12, characterized in that: The method further comprises: training the autoencoder according to the positive sample original image and the target loss function, and determining the cluster center of the sample feature vector of the positive sample original image stored in the decoding vector storage space in the autoencoder, comprising: Determining the vehicle model information of the positive sample original image; The positive sample original image and its vehicle model information are input into the autoencoder, and the autoencoder is trained according to the target loss function, wherein the decoding vector storage space is divided into independent decoding vector storage subspaces corresponding to each vehicle model information according to the vehicle model information of the positive sample original image, and the clustering center of the sample feature vector of the positive sample original image corresponding to each vehicle model information is stored in the decoding vector storage subspace.

18. The image processing method according to any one of claims 12 to 17, characterized in that: The decoding vector storage space comprises a decoding vector storage module, wherein the decoding vector storage module is used to store the cluster center of the sample feature vector in the space corresponding to the positive sample original image.

19. An image processing device, characterized in that: include: The original image acquisition unit to be tested is used to acquire the original image to be tested; A target feature map generating unit, used for encoding the original image to be tested to generate a target feature map of the original image to be tested; A first target decoding vector obtaining unit is used to obtain a first target decoding vector matching the target feature map according to a positive sample original image, wherein the positive sample original image is an image without abnormal regions, and comprises: searching for a clustering center of a sample feature vector of a positive sample original image matching the target feature map as a target clustering center from clustering centers of sample feature vectors of each positive sample original image, so as to generate the first target decoding vector; A generated image to be tested obtaining unit, used for decoding the first target decoding vector to obtain a generated image to be tested of the original image to be tested; A difference image generating unit, configured to generate a difference image according to the original image to be tested and the generated image to be tested; The abnormal region identification unit is used to identify the abnormal region in the original image to be tested according to the difference image.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 18 is implemented.

21. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method as described in any one of claims 1 to 18.

Images