Image processing method and device, computer equipment and storage medium

Abstract

The invention relates to the field of image processing, and discloses an image processing method and device, computer equipment and a storage medium, and the image processing method comprises the steps: obtaining image data; the image data comprises a previous decoding frame and a current frame to be compressed; extracting a previously decoded frame feature map from the previously decoded frame; extracting a current frame feature map from the current frame; determining an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; determining a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; determining a feature map residual error according to the current frame feature map and the current frame prediction feature map; and encoding the inter-frame motion vector and the feature map residual error to generate encoded data of the current frame. According to the method, global motion estimation is realized, the precision is ensured, the processing speed is high, and the encoding and decoding speed of image data is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, computer equipment and storage medium. Background Art

[0002] As the number of security cameras increases, the image data collected by these security cameras is also increasing. If these image data are transmitted directly without processing, a lot of bandwidth and storage space will be wasted, and a lot of processing time will also be consumed. Therefore, video compression technology is generally used to compress these image data.

[0003] Traditional video compression technologies, such as pixel-space motion compensation, discrete cosine transform (DCT), and quantization, can reduce spatial and temporal redundancy, but in the face of massive amounts of video, more efficient video compression technologies are needed. Moreover, the motion compensation and motion estimation of existing video compression technologies are both completed in pixel space, and global motion estimation consumes a lot of computing power. Although local motion estimation can reduce computing power consumption, it will significantly reduce accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, device, computer equipment and storage medium to address the above technical problems, so as to improve the speed of video encoding and decoding while ensuring the accuracy of the video.

[0005] An image processing method, comprising: Acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; Extracting a previously decoded frame feature map from the previously decoded frame; extracting a current frame feature map from the current frame; Determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; Determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; Determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The inter-frame motion vector and the feature map residual are encoded to generate encoded data of the current frame.

[0006] An image processing device, comprising: An image data acquisition module, used to acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; A feature map extraction module, used to extract a feature map of a previously decoded frame from the previously decoded frame; and to extract a feature map of a current frame from the current frame; A motion vector module, used to determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; A motion compensation module, used to determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; A residual calculation module, used to determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The encoding module is used to encode the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame.

[0007] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image processing method when executing the computer-readable instructions.

[0008] One or more readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the image processing method as described above.

[0009] The above-mentioned image processing method, device, computer equipment and storage medium, by performing motion estimation and motion compensation on the current frame in the feature space, not only realizes global motion estimation and ensures accuracy, but also has a fast processing speed, thereby greatly improving the encoding and decoding speed of image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a flow chart of an image processing method according to an embodiment of the present invention; Figure 2 is a structural schematic diagram of an image processing device in one embodiment of the present invention; Figure 3 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The image processing method provided in this embodiment can be applied to a security camera, or to a device for forwarding image data collected by a security camera. Figure 1 As shown, an image processing method is provided, comprising the following steps S10 to S60.

[0014] S10, acquiring image data; the image data includes a previously decoded frame and a current frame to be compressed.

[0015] It is understandable that the image data can be a digital signal. The security camera collects analog video signals, and after analog-to-digital conversion and color space conversion, it is converted into a digital signal that can be processed by a computer, that is, image data. If the current frame to be compressed is the tth frame, it is represented by X t , then the image data X can be expressed as: For the current frame X t When performing compression encoding, the previously decoded frame is needed. At time t, the image of the previous t-1 frame has been encoded and decoded, and the image of the decoded frame is represented as follows: The previously decoded frame is the reference frame closest to the current frame before the current frame, such as the first decoded frame before the current frame, that is, .

[0016] S20, extracting a previously decoded frame feature map from the previously decoded frame; and extracting a current frame feature map from the current frame.

[0017] Understandably, the feature map of the previously decoded frame can be extracted in the feature space to obtain the feature map of the previously decoded frame. Unlike the pixel space, the feature space is a low-dimensional representation obtained by extracting features from the image. The features in the feature map of the previously decoded frame can be regarded as a meaningful abstract representation of the previously decoded frame. These features can be high-level expressions such as edges, textures, and shapes of the image. The feature map of the previously decoded frame can reduce redundancy.

[0018] Similarly, the feature map of the current frame can be extracted in the feature space to obtain the feature map of the current frame. The features in the feature map of the current frame can be regarded as a meaningful abstract representation of the current frame.

[0019] Optionally, step S20, i.e., extracting a previously decoded frame feature map from the previously decoded frame; and extracting a current frame feature map from the current frame, comprises: S201, extracting features of the previously decoded frame in a feature space through an autoencoder network to obtain a feature map of the previously decoded frame; S202. Extract features of the current frame in a feature space through the autoencoder network to obtain a feature map of the current frame.

[0020] It is understandable that the features of the previous decoded frame in the feature space can be extracted by the auto-encoder network to obtain the feature map of the previous decoded frame. The auto-encoder network is a neural network model for unsupervised learning. The feature map can be extracted from the previous decoded frame by the encoder of the auto-encoder network, that is, the feature map of the previous decoded frame. It can be used as a reference frame for the current frame, thus decoding the feature map of the previous frame It can be expressed as: in, Indicates that the encoding is performed using an autoencoder network.

[0021] The features of the current frame in the feature space can be extracted by the autoencoder network to obtain the feature map of the current frame. The feature map can be extracted from the current frame by the encoder of the autoencoder network, that is, the feature map of the current frame. In one example, the feature map of the current frame It can be expressed as: This embodiment realizes the extraction of feature maps of the previously decoded frame and the current frame.

[0022] S30. Determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map.

[0023] Understandably, the process of obtaining inter-frame motion vectors is generally called motion estimation. The conventional pixel space motion estimation is to divide each frame of an image into a number of non-overlapping macroblocks and assume that the displacement of all pixels in the macroblocks is the same. The motion estimation of this embodiment is performed in the feature space.

[0024] Optionally, step S30, i.e., determining the inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map, includes: S301, dividing the current frame feature map into non-overlapping feature blocks in the feature space; the displacements of all pixels corresponding to the feature blocks are the same; S302, searching for the most similar feature block of the feature block within a specified search range in the previous frame decoding feature map according to a preset matching rule; S303: Determine the inter-frame motion vector according to the relative displacement between the feature block and the most similar feature block.

[0025] Understandably, dividing the feature map of the current frame into non-overlapping feature blocks means dividing the feature map according to a certain size, the pixels inside each feature block are continuous, and there is no overlap between different feature blocks.

[0026] The displacement of all pixels corresponding to the feature block is the same, which means that after the feature map of the current frame is processed by the autoencoder network, each feature block in the feature map of the current frame corresponds to a part of the current frame (original image), and these feature blocks have the same displacement. This means that no matter where a feature block appears in the current frame, other identical feature blocks will also appear in the corresponding feature map of the current frame with the same displacement. This displacement of feature blocks is generally caused by convolution operations. The convolution operation calculates on the feature map of the current frame by sliding a convolution kernel (filter), and each calculation corresponds to the features of a feature block. During the sliding process, the displacement of the convolution kernel is fixed, so the displacement of the pixels corresponding to the feature block is also the same. By having the same displacement of the feature blocks, the autoencoder network can capture similar features at different positions on the feature map. This way of sharing weights can improve the efficiency of the model, reduce the number of parameters, and help the model to be translationally invariant to the image.

[0027] For each feature block, the most similar feature block can be found in the specified search range of the previous frame decoded feature map according to the preset matching rule. The specified search range and the preset matching rule can be set according to actual needs. The relative displacement between the feature block and the most similar feature block can be calculated, and the relative displacement is the inter-frame motion vector. Here, the inter-frame motion vector can be a global motion vector.

[0028] This embodiment realizes the acquisition of inter-frame motion vectors, and the inter-frame motion vectors can be used to encode the current frame.

[0029] S40, determining a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map.

[0030] Understandably, after obtaining the inter-frame motion vector, the current frame prediction feature map can be obtained by motion compensation. Motion compensation is a method of describing the difference between adjacent frames. In the feature space, motion compensation describes how the feature block of the reference frame moves to a certain position in the current frame feature map. Through motion compensation, the predicted value of the image feature map of the current frame can be obtained, which is the current frame prediction feature map.

[0031] Optionally, step S40, i.e., determining the current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map, includes: S401. In a feature space, perform motion compensation on the previously decoded frame feature map according to the motion vector to generate the current frame prediction feature map.

[0032] Understandably, in the feature space, the feature block in the feature map of the previous decoded frame can be offset according to the motion vector to obtain the predicted value of the image feature map of the current frame, that is, the predicted feature map of the current frame. The predicted feature map of the current frame can be expressed as: in, Represents the current frame prediction feature map; Indicates motion compensation.

[0033] This embodiment is used to generate a current frame prediction feature map.

[0034] S50. Determine a feature map residual according to the current frame feature map and the current frame predicted feature map.

[0035] Understandably, the difference between the current frame feature map and the current frame predicted feature map can be calculated to generate a feature map residual.

[0036] Optionally, step S50, i.e., determining a feature map residual according to the current frame feature map and the current frame predicted feature map, includes: S501. In a feature space, determine a difference between the current frame feature map and the current frame predicted feature map as the feature map residual.

[0037] It is understandable that the traditional method calculates the residual in the pixel space, while the present embodiment calculates the feature map residual in the feature space. And the current frame prediction feature map The feature map residual can be calculated , that is: This embodiment is used to generate a feature map residual, which can be used to encode the current frame.

[0038] S60, encoding the inter-frame motion vector and the feature map residual to generate encoded data of the current frame.

[0039] Understandably, when compressing the current frame, only the inter-frame motion vector and the feature map residual need to be saved to restore the current frame. Therefore, the inter-frame motion vector and the feature map residual can be encoded respectively to generate the encoded data of the current frame. The encoded data is the compressed data of the current frame.

[0040] Optionally, the coded data of the current frame includes motion vector coding and residual coding; Step S60, i.e. encoding the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame, includes: S601, encoding the inter-frame motion vector to obtain the motion vector code; S602: Encode the feature map residual to obtain the residual coding.

[0041] Understandably, the inter-frame motion vector can be encoded to generate a motion vector encoding , that is: in, Entropy coding is used. Entropy coding can be Huffman coding.

[0042] The feature map residual can be encoded to generate residual coding , that is: This embodiment is used to generate motion vector coding and residual coding to achieve compression of the current frame.

[0043] In steps S10 to S60, by performing motion estimation and motion compensation on the current frame in the feature space, not only global motion estimation is achieved to ensure accuracy, but also the processing speed is fast, thereby greatly improving the encoding and decoding speed of image data.

[0044] Optionally, after step S60, that is, after encoding the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame, the method further includes: S70, decoding the encoded data to generate a current frame decoding feature map; S80, restoring the decoded feature map of the current frame to generate a restored image of the current frame.

[0045] Understandably, the encoded data can be decoded to generate the current frame decoding feature map , that is: in, The entropy decoding used is consistent with the probability model of the entropy coding in step S60.

[0046] Then restore the decoded feature map of the current frame to generate the restored image of the current frame , that is: in, The decoder of the autoencoding network used corresponds to the encoder used in step S202.

[0047] This embodiment realizes decoding of the encoded data, obtains a restored image of the current frame, and ensures high-precision restoration of the current frame.

[0048] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0049] In one embodiment, an image processing device is provided, which corresponds one-to-one to the image processing method in the above embodiment. Figure 2 As shown, the image processing device includes an image data acquisition module 10, a feature map extraction module 20, a motion vector module 30, a motion compensation module 40, a residual calculation module 50 and an encoding module 60. The functional modules are described in detail as follows: An image data acquisition module 10 is used to acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; The feature map extraction module 20 is used to extract a feature map of a previously decoded frame from the previously decoded frame; and to extract a feature map of a current frame from the current frame; A motion vector module 30, configured to determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; A motion compensation module 40, configured to determine a current frame prediction feature map based on the inter-frame motion vector and the previously decoded frame feature map; A residual calculation module 50, configured to determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The encoding module 60 is used to encode the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame.

[0050] Optionally, the image processing device further includes: A decoding module, used to decode the encoded data and generate a current frame decoding feature map; The image restoration module is used to restore the decoded feature map of the current frame to generate a restored image of the current frame.

[0051] Optionally, the feature map extraction module 20 includes: A previously decoded frame feature map extraction unit, configured to extract features of the previously decoded frame in a feature space through an autoencoder network to obtain the previously decoded frame feature map; The current frame feature map extraction unit is used to extract the features of the current frame in the feature space through the autoencoder network to obtain the current frame feature map.

[0052] Optionally, the motion vector module 30 includes: A feature block segmentation unit, used for segmenting the feature map of the current frame into non-overlapping feature blocks in the feature space; the displacements of all pixels corresponding to the feature blocks are the same; An approximate feature block search unit, configured to search for the most approximate feature block of the feature block within a specified search range in the previous frame decoding feature map according to a preset matching rule; The motion vector determining unit is used to determine the inter-frame motion vector according to the relative displacement between the feature block and the most similar feature block.

[0053] Optionally, the motion compensation module 40 includes: The current frame prediction feature map generating unit is used to perform motion compensation on the previously decoded frame feature map according to the motion vector in the feature space to generate the current frame prediction feature map.

[0054] Optionally, the residual calculation module 50 includes: A feature map residual unit is determined, which is used to determine the difference between the current frame feature map and the current frame predicted feature map as the feature map residual in the feature space.

[0055] Optionally, the coded data of the current frame includes motion vector coding and residual coding; the coding module 60 includes: A motion vector encoding unit, used for encoding the inter-frame motion vector to obtain the motion vector code; The residual coding unit is used to encode the feature map residual to obtain the residual coding.

[0056] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0057] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the image processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, an image processing method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0058] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the following steps are implemented: Acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; Extracting a previously decoded frame feature map from the previously decoded frame; extracting a current frame feature map from the current frame; Determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; Determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; Determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The inter-frame motion vector and the feature map residual are encoded to generate encoded data of the current frame.

[0059] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the following steps are implemented: Acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; Extracting a previously decoded frame feature map from the previously decoded frame; extracting a current frame feature map from the current frame; Determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; Determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; Determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The inter-frame motion vector and the feature map residual are encoded to generate encoded data of the current frame.

[0060] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0061] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0062] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; Extracting a previously decoded frame feature map from the previously decoded frame; extracting a current frame feature map from the current frame; Determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; Determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; Determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The inter-frame motion vector and the feature map residual are encoded to generate encoded data of the current frame.

2. The image processing method according to claim 1, characterized in that: After encoding the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame, the method further includes: Decoding the encoded data to generate a current frame decoding feature map; The decoded feature map of the current frame is restored to generate a restored image of the current frame.

3. The image processing method according to claim 1, characterized in that: The extracting of a previously decoded frame feature map from the previously decoded frame; and extracting a current frame feature map from the current frame include: Extracting features of the previously decoded frame in a feature space through an autoencoder network to obtain a feature map of the previously decoded frame; The features of the current frame in the feature space are extracted through the autoencoder network to obtain the current frame feature map.

4. The image processing method according to claim 1, wherein: The determining of the inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map comprises: In the feature space, the feature map of the current frame is divided into non-overlapping feature blocks; the displacements of all pixels corresponding to the feature blocks are the same; Find the most similar feature block of the feature block within a specified search range in the previous frame decoding feature map according to a preset matching rule; The inter-frame motion vector is determined according to the relative displacement between the feature block and the most similar feature block.

5. The image processing method according to claim 1, wherein: The determining the current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map comprises: In the feature space, motion compensation is performed on the previously decoded frame feature map according to the motion vector to generate the current frame prediction feature map.

6. The image processing method according to claim 1, characterized in that: The determining a feature map residual according to the current frame feature map and the current frame predicted feature map comprises: In the feature space, a difference between the current frame feature map and the current frame predicted feature map is determined as the feature map residual.

7. The image processing method according to claim 1, characterized in that: The coded data of the current frame includes motion vector coding and residual coding; The encoding of the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame includes: Encoding the inter-frame motion vector to obtain the motion vector code; The feature map residual is encoded to obtain the residual encoding.

8. An image processing device, characterized in that: include: An image data acquisition module, used to acquire image data; the image data includes a previously decoded frame and a current frame to be compressed; A feature map extraction module, used to extract a feature map of a previously decoded frame from the previously decoded frame; and to extract a feature map of a current frame from the current frame; A motion vector module, used to determine an inter-frame motion vector according to the previous frame decoding feature map and the current frame feature map; A motion compensation module, used to determine a current frame prediction feature map according to the inter-frame motion vector and the previously decoded frame feature map; A residual calculation module, used to determine a feature map residual according to the current frame feature map and the current frame predicted feature map; The encoding module is used to encode the inter-frame motion vector and the feature map residual to generate the encoded data of the current frame.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the image processing method according to any one of claims 1 to 7 is implemented.

10. One or more readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the image processing method according to any one of claims 1 to 7.

Images