Image processing method, device, electronic device and storage medium

By introducing an alignment module into the image processing model, the method of extracting and transferring the foreground area is solved, and the problem of large amount of image processing model calculation and high feature alignment error rate in autonomous driving is achieved, and more efficient feature alignment and accuracy are achieved.

CN114638856BActive Publication Date: 2024-11-26GUANGZHOU XIAOPENG CONNECTIVITY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210224411.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-09
Publication Date
2024-11-26
Estimated Expiration
2042-03-09

AI Technical Summary

Technical Problem

The existing feature alignment method based on image processing models is computationally expensive, which leads to excessive learning of the model and high feature alignment error rate, making it difficult to effectively process images of moving targets in autonomous driving.

Method used

By introducing an alignment module into the image processing model, the foreground area of ​​the past frame is extracted and its movement offset relative to the current frame is determined, and the position transfer process is performed to reduce the calculation amount and improve the accuracy of feature alignment.

Benefits of technology

It reduces the computational cost and learning difficulty of image processing models, improves the accuracy of feature alignment, and is suitable for visual processing tasks in autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114638856B_ABST
    Figure CN114638856B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an image processing method, device, electronic device and storage medium, the method comprising: inputting a current frame and a past frame into an alignment module; the past frame is an image frame before the current frame in time sequence; the alignment module is a processing module in an image processing model, and the image processing model is obtained by training using sample data; extracting a first foreground area of ​​the past frame through the alignment module; determining a movement offset of the first foreground area relative to the current frame through the alignment module; performing position transfer processing on the first foreground area according to the movement offset through the alignment module to obtain a second foreground area aligned with the position of the current frame. Implementing the embodiment of the present application can reduce the amount of calculation when the image processing model performs feature alignment operations, thereby reducing the learning difficulty of the model and improving the accuracy of feature alignment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Vision-based autonomous driving solutions often require processing of images captured by the vehicle to convert the information in the images into data that can be understood by the vehicle's processor. Vehicles generally continuously capture surrounding road conditions while driving, so both the foreground and background areas of the captured images will appear in motion. At this point, it is generally necessary to align the features of moving targets in continuous images to facilitate subsequent image processing operations.

[0003] In the prior art, neural networks, classifiers and other image processing models can be trained through machine learning methods, and the trained models can be used to perform feature alignment on moving targets in continuous images. However, in practice, it is found that the current feature alignment methods based on image processing models have a large amount of computation, which can easily lead to problems such as excessive difficulty in model learning and high feature alignment error rate. Summary of the invention

[0004] The embodiments of the present application disclose an image processing method, device, electronic device and storage medium, which can reduce the amount of calculation when the image processing model performs feature alignment operations, thereby reducing the learning difficulty of the model and improving the accuracy of feature alignment.

[0005] The present application provides an image processing method, the method comprising:

[0006] Input the current frame and the past frame into an alignment module; the past frame is an image frame that is before the current frame in time sequence; the alignment module is a processing module in an image processing model, and the image processing model is obtained by training using sample data;

[0007] extracting a first foreground region of the past frame by the alignment module;

[0008] Determining, by the alignment module, a movement offset of the first foreground area relative to the current frame;

[0009] The alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position.

[0010] In one embodiment, extracting the first foreground area of ​​the past frame by the alignment module includes:

[0011] identifying, by the alignment module, a foreground mask of the past frame with reference to the current frame;

[0012] The alignment module uses the foreground mask to extract a first foreground area from the past frame.

[0013] In one embodiment, identifying the first foreground area of ​​the past frame by the alignment module with reference to the current frame includes:

[0014] Processing the current frame and the past frame based on an optical flow method to identify a foreground mask of the past frame;

[0015] Alternatively, the alignment module includes: a classification layer; the identifying the foreground mask of the past frame by the alignment module with the current frame as a reference includes:

[0016] The past frame and the current frame are processed by the classification layer to identify a foreground mask of the past frame.

[0017] In one embodiment, the alignment module includes: a convolution layer; and determining the movement offset of the first foreground area relative to the current frame by the alignment module includes:

[0018] Performing tensor concatenation on the first foreground area and the current frame, and inputting the concatenated image obtained by concatenating the tensors into the convolution layer; the convolution layer is obtained by training using the sample data;

[0019] The series of images are processed by the convolution layer to determine a movement offset of the first foreground area relative to the current frame.

[0020] In one embodiment, after inputting the current frame and the past frame into the alignment module, the method further comprises:

[0021] Extracting a first background region of the past frame by the alignment module;

[0022] Furthermore, after the alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position, the method further includes:

[0023] Suppressing the pixels in the first background area that are covered by the second foreground area by the alignment module to obtain a second background area;

[0024] The second foreground area and the second background area are merged by the alignment module to obtain a feature map aligned with the current frame.

[0025] In one embodiment, suppressing the pixels in the first background area that are covered by the second foreground area by the alignment module to obtain the second background area includes:

[0026] Determining, by the alignment module, the pixel points in the first foreground area that are covered by the second foreground area according to the image position of each background pixel point included in the first background area and the image position of each foreground pixel point included in the second foreground area;

[0027] The alignment module clears pixel values ​​corresponding to covered pixels in the first background area to obtain a second background area.

[0028] In one embodiment, the image processing model further includes: a timing loop module and a delay module; and the step of inputting the current frame and the past frame into the alignment module includes:

[0029] After the timing cycle module processes the input first characteristic graph to obtain a second characteristic graph, the second characteristic graph output by the timing cycle module at the first moment is input into the delay module;

[0030] Input the second feature map output by the delay module at the second moment as a past frame to the alignment module, and input the current frame obtained at the second moment to the alignment module; the second moment is after the first moment in terms of timing;

[0031] Furthermore, after the alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position, the method further includes:

[0032] The second foreground area is input into the timing loop module as a first feature map.

[0033] The present application discloses an image processing device, including:

[0034] An input unit, used to input a current frame and a past frame into an alignment module; the past frame is an image frame that is before the current frame in time sequence; the alignment module is a processing module in an image processing model, and the image processing model is obtained by training using sample data;

[0035] an extraction unit, configured to extract a first foreground area of ​​the past frame through the alignment module;

[0036] a determining unit, configured to determine, by means of the alignment module, a movement offset of the first foreground area relative to the current frame;

[0037] The displacement unit is used to perform position shift processing on the first foreground area according to the movement offset through the alignment module to obtain a second foreground area aligned with the current frame position.

[0038] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements any one of the image processing methods disclosed in the embodiment of the present application.

[0039] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any image processing method disclosed in the embodiment of the present application is implemented.

[0040] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0041] Through the alignment module of the image processing model, the first foreground area of ​​the past frame can be extracted, and the movement offset of the first foreground area relative to the current frame can be determined, so that the first foreground area can be transferred based on the movement offset to obtain a second alignment area aligned with the current frame position. In other words, the alignment module can enable the image processing model to focus on calculating the movement offset of the foreground area, rather than directly calculating the offset for all areas in the image, thereby reducing the amount of calculation of the image processing model, reducing the difficulty of learning the model, and improving the accuracy of feature alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1A is a schematic diagram of the structure of a classic circulation module disclosed in an embodiment;

[0044] Figure 1B is a structural schematic diagram of an image processing model including an alignment module disclosed in an embodiment;

[0045] Figure 2 is a schematic diagram of a method flow of an image processing method disclosed in an embodiment;

[0046] Figure 3 is a method flow diagram of another image processing method disclosed in an embodiment;

[0047] Figure 4is an exemplary diagram of another image processing method disclosed in an embodiment;

[0048] Figure 5 is a structural schematic diagram of an image processing device disclosed in an embodiment;

[0049] Figure 6 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0052] The embodiments of the present application disclose an image processing method, device, electronic device and storage medium, which can reduce the amount of calculation when the image processing model performs feature alignment operations, thereby reducing the learning difficulty of the model and improving the accuracy of feature alignment. Detailed descriptions are given below.

[0053] First, the image processing model disclosed in the embodiment of the present application is introduced. The image processing model disclosed in the embodiment of the present application may include an alignment module and any existing neural network module. The alignment module is used to perform feature alignment on the current frame and the past frame. The current frame and the past frame may be any two frames captured by the camera device of the vehicle that are associated in time sequence, and the past frame is before the current frame in time sequence.

[0054] In some embodiments, any of the above-mentioned existing neural network modules may include a timing loop module and a delay module. The timing loop module may be a network module such as RNN, LSTM, GRU or their variants, and the delay module may be used to delay the input image frame by one or more time steps before outputting it. In other words, the image processing model disclosed in the embodiment of the present application may add an additional alignment module to the existing neural network structure, and the alignment module may be inserted into any existing video detection and target recognition model without modifying most of the model structure of the existing neural network model.

[0055] For example, see Figure 1A , Figure 1A FIG. 1 is a schematic diagram of a structure of a classic cycle module disclosed in an embodiment. Figure 1A As shown, the classic loop structure may include a delay module 110 and a timing loop module 120 .

[0056] in, Figure 1A The x shown t is the characteristic graph at time t, h t-1 is the characteristic graph at time t-1, and t can be a positive integer greater than or equal to 1. That is, x t Can be used as the current frame, h t-1 It can be a past frame and can be output by the delay unit 110.

[0057] In the classic loop module, the current frame x t and the past frame h t-1 can be input to the timing cycle module 120, and the timing cycle model 120 processes the current frame x t and the past frame h t-1 Perform one or more image processing operations such as target detection, target extraction, target recognition, and output feature map h t , feature map h t It can be further input to the delay unit 110, which can output the feature map h after delaying the step length of a time step (Time Step). t , as the past frame at time t+1.

[0058] See also Figure 1B , Figure 1B FIG. 1 is a schematic diagram of a structure of an image processing model including an alignment module disclosed in an embodiment. Figure 1B As shown, it can be Figure 1A As shown, an alignment module 130 is inserted between the delay module 110 and the timing cycle module 120 .

[0059] exist Figure 1B In the image processing module shown in FIG. 1 , the feature map h output by the delay unit 110 is t-1 can be input into the alignment module 130 as the past frame, and the current frame x t It can also be input to the alignment module 130. The alignment module 130 can t and the past frame h t-1 Perform feature alignment to obtain the aligned feature map h′ t-1 The feature map h′ after feature alignment t-1 and the current frame x tIt can be further input into the timing cycle module 120, and the timing cycle module 120 performs feature alignment on the feature map h′ t-1 and the current frame x t Perform one or more image processing operations such as target detection, target extraction, and target recognition.

[0060] The above is an example of an image processing model including an alignment module disclosed in an embodiment of the present application. It should be noted that, in some possible embodiments, the image processing model may also be used only to perform feature alignment tasks and may only include an alignment module.

[0061] Based on the aforementioned image processing model, the following content introduces the image processing method disclosed in the embodiment of the present application. Figure 2 , Figure 2 The method flow chart of an image processing method disclosed in an embodiment is as follows. The method can be applied to any electronic device storing the aforementioned image processing model, such as a vehicle processor, a personal computer, etc., without specific limitation. Figure 2 As shown, the image processing method may include the following steps:

[0062] 210. Input the current frame and the past frame into the alignment module.

[0063] In some embodiments, the current frame and the past frame may be any two image frames captured by a camera of the vehicle and associated in time sequence.

[0064] In some other possible embodiments, the current frame and the past frame may also be feature maps output by other processing modules in the image processing model except the alignment module. For example, the current frame may be a feature map output by an upstream processing module of the alignment module, and the past frame may be a feature map output by a delay module in the image processing model.

[0065] For example, please refer to the above Figure 1B At the first moment (t-1), the timing cycle module may output a second feature map, which is obtained after the timing cycle module processes the input first feature map. The first feature map may include Figure 1B The alignment module shown in the figure outputs the feature map h′ after feature alignment at time t-2 t-2 , and the current frame x corresponding to the first moment (t-1 moment) t-1 .

[0066] After waiting for a time step, the delay module can output the second feature map at the second time (time t). The second feature map output by the delay module at the second time (time t) can be used as the past frame h corresponding to time t. t-1 .

[0067] 220. Extract the first foreground area of ​​the past frame through the alignment module.

[0068] In the application scenarios of autonomous driving, the foreground area generally includes cars, bicycles, pedestrians and other targets that vehicles need to pay attention to or avoid, while the background area generally includes roads, buildings, trees, etc.

[0069] In some embodiments, the alignment module may detect and identify the past frame itself to extract the first foreground area. For example, the alignment module may identify objects such as cars, bicycles, pedestrians, etc. in the past frame to identify the image area of ​​the objects such as cars, bicycles, pedestrians, etc. in the past frame as the first foreground area; or the alignment module may also identify the depth of the pixel points in the past frame to identify the image area with a depth value less than a depth threshold as the first foreground area.

[0070] In other embodiments, the alignment module uses the current frame as a reference and extracts the first foreground area from the past frame in combination with the current frame. For example, objects in the foreground area, such as cars, bicycles, and pedestrians, are often moving, while objects in the background area, such as buildings and roads, are often stationary. Therefore, the position offset of the objects in the foreground area between the current frame and the past frame may be smaller than the position offset of the objects in the background area between the current frame and the past frame. The alignment module can identify the first foreground area of ​​the past frame by comparing the current frame with the past frame.

[0071] The alignment module may first identify the foreground mask of the past frame with the current frame as a reference, and the foreground mask may be used to indicate the image position of the first foreground region in the past frame. Therefore, the alignment module may further use the foreground mask to extract the first foreground region from the past frame.

[0072] Exemplarily, the alignment module extracts the first foreground area from the past frame, which can be expressed by the following formula:

[0073] h fore =mask fore ·h t-1 ;Formula (1)

[0074] Among them, h fore Can be used to represent the first foreground area, mask fore Can be used to represent the foreground mask, h t-1 Can be used to represent past frames.

[0075] As an optional implementation, the alignment module may include a classification layer. The classification layer may be a network layer in a neural network, and may be implemented by a classification function such as a Sigmoid function, a Softmax function, etc. The classification layer may include one or more learning parameters, and the process of training the image processing model using sample data may include adjusting each learning parameter in the classification layer. At the end of the training of the image processing model, the learning parameters in the classification layer enable the classification layer to accurately identify the foreground mask of the past frame with the current frame as a reference.

[0076] Exemplarily, the operation of the alignment module processing the past frame and the current frame through the classification layer to identify the foreground mask of the past frame can be expressed by the following formula:

[0077] mask fore =σ(U1x t +W1h t-1 +b1);Formula (2);

[0078] Among them, mask fore can be used to represent the foreground mask, σ(·) can be used to represent the Sigmoid function, x t Can be used to indicate the current frame, h t-1 It can be used to represent the past frame; U1 and W1 can be used to represent the learning parameters in the classification layer, and b1 can be used to represent the bias term.

[0079] As another optional implementation, the alignment module may also identify the foreground mask of the past frame by a non-machine learning method. For example, the past frame and the current frame may be processed by an optical flow method to identify the foreground mask of the past frame. The optical flow method is a method that uses the change of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0080] Therefore, the alignment module can identify the correspondence between the current frame and the past frame through the optical flow method, thereby identifying the moving object between the current frame and the past frame, and determining the foreground mask of the past frame according to the image position of the moving object in the past frame.

[0081] Exemplarily, the alignment module processes the past frame and the current frame by an optical flow method to identify the foreground mask of the past frame, which can be expressed by the following formula:

[0082] mask fore =optical_flow(h t-1 ,x t ); formula (3);

[0083] Among them, mask forecan be used to represent foreground masks, optical_flow(·) can be used to represent optical flow, x t Can be used to indicate the current frame, h t-1 Can be used to represent past frames.

[0084] 230. Determine a movement offset of the first foreground area relative to the current frame through an alignment module.

[0085] In the embodiment of the present application, the offset of the first foreground area relative to the current frame refers to the position offset of each pixel point of the foreground area in the previous frame relative to the corresponding pixel point in the current frame.

[0086] As an optional implementation, the alignment module may include a convolution layer, which may be a network layer in a neural network and may be used to perform a convolution operation. The convolution layer may also include one or more learning parameters, and the process of training the image processing model using sample data may also include adjusting each learning parameter in the convolution layer. At the end of the training of the image processing model, the learning parameters in the convolution layer enable the convolution layer to accurately calculate the position offset of each pixel point in the foreground area in the past frame relative to the corresponding pixel point in the current frame.

[0087] Therefore, the alignment module can first extract the first foreground area from the past frame and perform tensor concatenation with the current frame to obtain a concatenated image; then input the concatenated image into the convolution layer, and process the concatenated image through the convolution layer to determine the movement offset of the first foreground area relative to the current frame.

[0088] Exemplarily, the movement offset of the first foreground area relative to the current frame determined by the alignment module can be expressed by the following formula:

[0089] offset=Conv(Concat([h fore ,x t ])); Formula (4);

[0090] Among them, offset can be used to represent the movement offset, Conv(·) can be used to represent the convolution layer, Concat(·) can be used to represent the operation of tensor concatenation, and h fore Can be used to represent the first foreground area, x t Can be used to indicate the current frame.

[0091] 240. Perform position transfer processing on the first foreground area according to the movement offset through an alignment module to obtain a second foreground area aligned with the current frame position.

[0092] In the embodiment of the present application, performing a position transfer process on the first foreground area according to the movement offset may refer to performing a position transfer on each pixel in the first foreground area according to the movement offset through operations such as affine transformation, interpolation operation, distortion operation, etc. The image position of the second foreground area obtained after the transfer is the same as or similar to the image position of the image content included in the second foreground area in the current frame, that is, the second foreground area is aligned with the current frame position.

[0093] Exemplarily, the operation of the alignment module performing position transfer processing on the first foreground area according to the movement offset can be expressed by the following formula:

[0094] h′ fore =Warp(h fore ,offset); formula (5);

[0095] Among them, h′ fore can be used to represent the second foreground area, Warp(·) can be used to represent the warping operation, h fore It can be used to indicate the first foreground area, and offset can be used to indicate the movement offset.

[0096] In summary, based on the image processing model including the alignment module disclosed in the aforementioned embodiment, and the image processing method disclosed in the aforementioned embodiment, the foreground area can be accurately identified, and the image processing model can be made to focus more on the offset calculation of the foreground area, rather than directly performing offset calculations on all areas in the image, thereby reducing the amount of calculation of the image processing model, reducing the learning difficulty of the model, and improving the accuracy of feature alignment. In addition, the alignment module can be easily inserted into any existing image processing model, with little modification to the original network structure of the image processing model, and can achieve a plug-and-play effect.

[0097] In addition, in application scenarios such as autonomous driving, the foreground area is actually more important, and the importance of the background area is relatively lower than that of the foreground area. Therefore, even if only the offset of the foreground area is calculated and the foreground area of ​​the current frame and the past frame are aligned, the safety image of the vision-based autonomous driving solution is low.

[0098] See also Figure 3 , Figure 3 The method flow chart of another image processing method disclosed in an embodiment is as follows. The method can be applied to any electronic device storing the aforementioned image processing model. Figure 3 As shown, the image processing method may include the following steps:

[0099] 310. Input the current frame and the past frame into the alignment module.

[0100] 320. Extract a first foreground area and a first background area of ​​a past frame through an alignment module.

[0101] In the embodiment of the present application, the alignment module can not only extract the foreground area of ​​the past frame, but also extract the background area of ​​the past frame.

[0102] As an optional implementation, after extracting the first foreground area, the alignment module may determine the remaining image areas except the first foreground area in the past frame as the first background area.

[0103] The alignment module may first identify the foreground mask of the past frame with the current frame as a reference, and extract the first foreground region from the past frame using the foreground mask. Accordingly, the alignment module may invert the foreground mask to obtain the background mask of the past frame, and the background mask may be used to indicate the image position of the background region in the past frame, so that the first background region may be extracted from the past frame using the background mask.

[0104] Exemplarily, the alignment module extracts the first background area from the past frame, which can be expressed by the following formula:

[0105] h back =(1-mask fore )·h t-1 ; Formula (6);

[0106] Among them, h back Can be used to represent the first background area, mask fore Can be used to represent the foreground mask, h t-1 Can be used to represent past frames.

[0107] 330. Determine a movement offset of the first foreground area relative to the current frame through an alignment module.

[0108] 340. Perform position transfer processing on the first foreground area according to the movement offset through an alignment module to obtain a second foreground area aligned with the current frame position.

[0109] The implementation of step 330-step 340 can refer to the aforementioned step 230-step 240, and will not be repeated in the following content.

[0110] 350. Suppress the pixels in the first background area that are covered by the second foreground area through an alignment module to obtain a second background area.

[0111] In the embodiment of the present application, suppressing the pixel points in the first background area that are covered by the second foreground area may refer to reducing or clearing the pixel values ​​corresponding to the pixel points in the first background area that are covered by the second foreground area.

[0112] As an optional implementation, the alignment module can determine the pixel points in the first foreground area that are covered by the second foreground area based on the image positions of the background pixel points included in the first background area and the image positions of the foreground pixel points included in the second foreground area. For example, if a background pixel point has the same image position as a foreground pixel point, then the background pixel point is a pixel point in the first background area that is covered by the second background area. Furthermore, the alignment module can clear the pixel values ​​corresponding to the covered pixel points in the first background area to obtain the second background area.

[0113] Exemplarily, the operation of the alignment module to suppress the pixels in the first background area that are covered by the second foreground area can be expressed by the following formula:

[0114] mask′ fore =Warp(mask fore ,offset); Formula (7);

[0115] h′ back =(1-mask′ fore )·h back ; Formula (8);

[0116] Among them, h′ back Can be used to represent the second background area, h back can be used to represent the first background area, Warp(·) can be used to represent the warping operation, and h fore It can be used to represent the first foreground area, and offset can be used to represent the movement offset. fore It can be used to represent the foreground mask after position shift, which can be equivalent to the second foreground area.

[0117] 360. Merge the second foreground area and the second background area through an alignment module to obtain a feature map aligned with the current frame.

[0118] In the embodiment of the present application, merging the second foreground area and the second background area may refer to superimposing the second foreground area and the second background area, and the image obtained after the superposition is a feature map aligned with the current frame.

[0119] Exemplarily, the operation of merging the second foreground area and the second background area can be expressed by the following formula:

[0120] h′ t-1 =h′ fore +h′ back ; Formula (9);

[0121] Among them, h′ t-1 Can be used to represent the feature map obtained after superposition, h′ foreCan be used to represent the second foreground area, h′ back Can be used to represent a second background area.

[0122] It can be seen that in the aforementioned embodiment, after calculating the movement offset of the foreground area between the current frame and the past frame, the alignment module can further align the foreground area and the background area of ​​the current frame and the past frame according to the movement offset, which can improve the accuracy of feature alignment. If subsequent image processing operations such as target detection and target recognition are performed based on the feature map obtained after feature alignment, the feature map in which the foreground area and the background area are aligned can improve the performance of subsequent image processing operations.

[0123] In order to more clearly illustrate the image processing method disclosed in the embodiment of the present application. Figure 4 , Figure 4 is an example diagram of another image processing method disclosed in an embodiment. Figure 4 As shown, first, the alignment module can be based on the current frame x t and the past frame h t-1 Distinguish the foreground area and the background area, and get the first foreground area h of the past frame fore and the first background area h back Then, according to the first foreground area h fore Calculate the movement offset offset. The movement offset offset can be used to perform position shift processing on the first foreground area to obtain the second foreground area h′ fore ; At the same time, the first background area h is suppressed by using the movement offset offset back The second background area h′ is obtained by back Finally, merge the second foreground region h′ fore and the second background region h′ back , get the feature map h′ aligned with the current frame feature t-1 .

[0124] In some embodiments, if the image processing model includes a sequential loop module, the feature map h′ output by the alignment module t-1 Can be compared with the current frame x at time t t As the first feature map input to the timing cycle module, the timing cycle module processes the feature map h′ t-1 and the current frame x t to be processed.

[0125] In addition, in some other embodiments, if the alignment module outputs the second foreground region h′ fore , the second foreground area h′ fore and the current frame x at time t tAs the first feature map input to the temporal cycle module, the temporal cycle module generates a second foreground region h′ fore and the current frame x t to be processed.

[0126] See also Figure 5 , Figure 5 The structure diagram of an image processing device disclosed in an embodiment is a schematic diagram, and the image processing device can be applied to any of the above-mentioned electronic devices. Figure 5 As shown, the image processing apparatus 500 may include: an input unit 510 , an extraction unit 520 , a determination unit 530 and a displacement unit 540 .

[0127] The input unit 510 can be used to input the current frame and the past frame into the alignment module; the past frame is an image frame that is before the current frame in time sequence; the alignment module is a processing module in the image processing model, and the image processing model is obtained by training using sample data;

[0128] An extraction unit 520, which can be used to extract a first foreground area of ​​a past frame through an alignment module;

[0129] A determination unit 530, which may be used to determine a movement offset of the first foreground area relative to the current frame through an alignment module;

[0130] The displacement unit 540 may be used to perform position shift processing on the first foreground area according to the movement offset through the alignment module to obtain a second foreground area aligned with the current frame position.

[0131] In one embodiment, the extraction unit 520 may also be used to identify a foreground mask of a past frame using an alignment module with the current frame as a reference; and extract the first foreground area from the past frame using the foreground mask using the alignment module.

[0132] Optionally, the extraction unit 520 may also be configured to process the current frame and the past frame based on an optical flow method to identify a foreground mask of the past frame.

[0133] Optionally, the alignment module may include a classification layer; the extraction unit 520 may also be used to process the past frame and the current frame through the classification layer to identify the foreground mask of the past frame.

[0134] In one embodiment, the extraction unit 520 may also be used to extract a first background region of a past frame through an alignment module;

[0135] The displacement unit 540 may also be used to suppress the pixel points in the first background area that are covered by the second foreground area through the alignment module to obtain the second background area;

[0136] Furthermore, the image processing device 500 may further include: a merging unit.

[0137] The merging unit may be used to merge the second foreground area and the second background area through the alignment module to obtain a feature map aligned with the current frame.

[0138] In one embodiment, the displacement unit 540 can also be used to determine the pixel points in the first foreground area that are covered by the second foreground area according to the image position of each background pixel point included in the first background area and the image position of each foreground pixel point included in the second foreground area through an alignment module; and to clear the pixel values ​​corresponding to the covered pixel points in the first background area through the alignment module to obtain the second background area.

[0139] In one embodiment, the image processing model further includes: a timing loop module and a delay module.

[0140] The input unit 510 can also be used to process the input first feature map in the timing loop module to obtain the second feature map, and then input the second feature map output by the timing loop module at the first moment into the delay module; and input the second feature map output by the delay module at the second moment into the alignment module as a past frame, and at the same time input the current frame obtained at the second moment into the alignment module; the second moment is after the first moment in terms of timing.

[0141] The input unit 510 can also be used to input the second foreground area as the first feature map into the timing loop module after the displacement unit 540 performs position transfer processing on the first foreground area according to the movement offset through the alignment module to obtain the second foreground area aligned with the current frame position.

[0142] It can be seen that by implementing the aforementioned image processing device, the foreground area can be accurately identified through an image processing model including an alignment module, and the image processing model can be made to focus more on the offset calculation of the foreground area, rather than directly performing offset calculations on all areas in the image. This can reduce the amount of calculation of the image processing model, reduce the difficulty of learning the model, and improve the accuracy of feature alignment.

[0143] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device disclosed in an embodiment. Figure 6 As shown, the electronic device may include:

[0144] A memory 610 storing executable program codes;

[0145] a processor 620 coupled to the memory 610;

[0146] The processor 620 calls the executable program code stored in the memory 610 to execute any image processing method disclosed in the embodiments of the present application.

[0147] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute any one of the image processing methods disclosed in the embodiment of the present application.

[0148] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute any one of the image processing methods disclosed in the embodiments of the present application.

[0149] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0150] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0151] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0154] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0155] The above is a detailed introduction to the image processing method, device, electronic device and storage medium disclosed in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Input the current frame and the past frame into an alignment module; the past frame is an image frame that is before the current frame in time sequence; the alignment module is a processing module in an image processing model, and the image processing model is obtained by training using sample data; extracting a first foreground region of the past frame by the alignment module; Determine the movement offset of the first foreground area relative to the current frame by the alignment module; the movement offset is used to represent the position offset of each pixel point of the first foreground area of ​​the past frame relative to the corresponding pixel point in the current frame; The alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position, so that the image position of the second foreground area is the same as or similar to the image position of the image content included in the second foreground area in the current frame.

2. The method according to claim 1, characterized in that: The extracting the first foreground area of ​​the past frame by the alignment module comprises: identifying, by the alignment module, a foreground mask of the past frame with reference to the current frame; The alignment module uses the foreground mask to extract a first foreground area from the past frame.

3. The method according to claim 2, characterized in that The step of identifying the foreground mask of the past frame by using the alignment module with reference to the current frame includes: Processing the current frame and the past frame based on an optical flow method to identify a foreground mask of the past frame; Alternatively, the alignment module includes: a classification layer; the identifying the foreground mask of the past frame by the alignment module with the current frame as a reference includes: The past frame and the current frame are processed by the classification layer to identify a foreground mask of the past frame.

4. The method according to claim 1, characterized in that: The alignment module includes: a convolution layer; and determining the movement offset of the first foreground area relative to the current frame by the alignment module includes: Performing tensor concatenation on the first foreground area and the current frame, and inputting the concatenated image obtained by concatenating the tensors into the convolution layer; the convolution layer is obtained by training using the sample data; The series of images are processed by the convolution layer to determine a movement offset of the first foreground area relative to the current frame.

5. The method according to any one of claims 1 to 4, characterized in that: After inputting the current frame and the past frame into the alignment module, the method further includes: Extracting a first background region of the past frame by the alignment module; Furthermore, after the alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position, the method further includes: Suppressing the pixels in the first background area that are covered by the second foreground area by the alignment module to obtain a second background area; The second foreground area and the second background area are merged by the alignment module to obtain a feature map aligned with the current frame.

6. The method according to claim 5, characterized in that The step of suppressing the pixel points in the first background area that are covered by the second foreground area by the alignment module to obtain the second background area includes: Determining, by the alignment module, the pixel points in the first foreground area that are covered by the second foreground area according to the image position of each background pixel point included in the first background area and the image position of each foreground pixel point included in the second foreground area; The alignment module clears pixel values ​​corresponding to covered pixels in the first background area to obtain a second background area.

7. The method according to any one of claims 1 to 4, characterized in that: The image processing model further includes: a timing loop module and a delay module; and the step of inputting the current frame and the past frame into the alignment module includes: After the timing cycle module processes the input first characteristic graph to obtain a second characteristic graph, the second characteristic graph output by the timing cycle module at the first moment is input to the delay module; Input the second feature map output by the delay module at the second moment as a past frame to the alignment module, and input the current frame obtained at the second moment to the alignment module; the second moment is after the first moment in terms of timing; Furthermore, after the alignment module performs position transfer processing on the first foreground area according to the movement offset to obtain a second foreground area aligned with the current frame position, the method further includes: The second foreground area is input into the timing loop module as a first feature map.

8. An image processing device, characterized in that: include: An input unit, used for inputting the current frame and the past frame into the alignment module; The past frame is an image frame that is before the current frame in time sequence; The alignment module is a processing module in the image processing model, and the image processing model is trained using sample data; an extraction unit, configured to extract a first foreground area of ​​the past frame through the alignment module; a determining unit, configured to determine, by means of the alignment module, a movement offset of the first foreground area relative to the current frame, wherein the movement offset is used to characterize a position offset of each pixel point of the first foreground area of ​​the past frame relative to a corresponding pixel point in the current frame; A displacement unit is used to perform position transfer processing on the first foreground area according to the movement offset through the alignment module to obtain a second foreground area aligned with the current frame position, so that the image position of the second foreground area is the same as or similar to the image position of the image content included in the second foreground area in the current frame.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.