Image processing method and device, terminal equipment and storage medium

Abstract

The application is suitable for the technical field of computer vision, and provides an image processing method and device, terminal equipment and a storage medium, which comprise the following steps: determining gradient data of a current image frame of a video sequence, calculating a gradient direction histogram feature of the current image frame according to the gradient data to obtain a first feature, determining a difference degree of the first feature and a second feature, if the difference degree is greater than a preset threshold, performing correction processing on the first feature to obtain a corrected first feature, the second feature is a gradient direction histogram feature of a previous image frame of the current image frame in the video sequence, performing detection on the first feature to obtain a detection result, and the detection result is used to indicate whether the current image frame has a detection target and position information of the detection target when the detection target exists. The application can improve the detection accuracy of target tracking.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to image processing methods, apparatus, terminal equipment and computer-readable storage media. Background Technology

[0002] Computer vision, also known as machine vision, is a science that studies how machines "see." Target tracking is one of the hottest research areas in computer vision. Camera tracking and focusing, automatic target tracking in drones, and other applications all require target tracking technology. Tracking specific objects is also closely related to our daily lives, such as vehicle tracking in security, gesture tracking in intelligent interactive systems, and face tracking.

[0003] In simple terms, target tracking is a task that involves giving an initial position of a target in a continuous video sequence and then continuously tracking and locating the target in each subsequent frame; it is equivalent to locating a moving target in a video.

[0004] Currently, target tracking is usually achieved by extracting texture or color features from images and then detecting the obtained image features. However, since the target is usually moving in the target tracking task, there may be problems such as changes in lighting and target deformation in each image frame of the video sequence, which can lead to the loss of the target during target tracking and result in poor target tracking performance. Summary of the Invention

[0005] This application provides an image processing method, apparatus, terminal device, and storage medium, which can improve the detection accuracy of target tracking.

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Determine the gradient data of the current image frame in the video sequence;

[0008] The gradient direction histogram feature of the current image frame is calculated based on the gradient data to obtain the first feature;

[0009] Determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, then perform a correction process on the first feature to obtain the corrected first feature. The second feature is the gradient orientation histogram feature of the previous image frame of the current image frame in the video sequence.

[0010] The first feature is detected to obtain a detection result, which is used to indicate whether the current image frame has a detection target and the location information of the detection target when the detection target exists.

[0011] Secondly, embodiments of this application provide an image processing apparatus, including:

[0012] The gradient calculation module is used to determine the gradient data of the current image frame in the video sequence;

[0013] The first feature acquisition module is used to calculate the gradient direction histogram feature of the current image frame based on the gradient data to obtain the first feature;

[0014] The correction module is used to determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, the first feature is corrected to obtain the corrected first feature. The second feature is the gradient orientation histogram feature of the previous image frame of the current image frame in the video sequence.

[0015] The detection module is used to detect the first feature and obtain a detection result. The detection result is used to indicate whether the current image frame has a detection target and the location information of the detection target when the detection target exists.

[0016] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image processing method described in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image processing method described in the first aspect above.

[0018] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the image processing method described in any one of the first aspects.

[0019] The beneficial effects of the embodiments in this application compared with the prior art are:

[0020] In this embodiment, since gradient histogram features can reflect the edge and corner information of objects in an image, thus reflecting the shape of the object well, and since gradient histogram features are not sensitive to changes in illumination, detection based on the obtained first feature, i.e., the gradient histogram feature of the current image frame of the video sequence, can better detect objects present in the current image frame, thereby accurately detecting the existing target and improving detection accuracy. Simultaneously, to reduce the differences between image frames in the video sequence caused by the movement of the target, after obtaining the first feature of the current image frame of the video sequence, it is determined whether the difference between the first feature and the second feature of the previous image frame is within a preset threshold. If the difference is greater than the preset threshold, the first feature is corrected to reduce the difference between the first and second features, facilitating the detection and tracking of specific targets in the current image frame, thereby improving target tracking efficiency. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0022] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0024] Figure 3 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0030] Example 1:

[0031] Figure 1 A schematic flowchart of an image processing method provided by an embodiment of the present invention is shown, and is described in detail below:

[0032] Step S101: Determine the gradient data of the current image frame of the video sequence.

[0033] Optionally, the aforementioned video sequence refers to the sequence of image frames corresponding to the video stream for which target tracking and detection are required; that is, a series of consecutive image frames constituting the video, and the video sequence includes at least two image frames. The aforementioned video sequence can be a sequence of image frames acquired in real time or a sequence of image frames acquired historically, depending on the user's actual application scenario. For example, in traffic management applications, the target vehicle is typically detected and tracked in real time using acquired video images to determine whether the target vehicle is engaging in speeding or other violations.

[0034] Specifically, since the gradient data obtained by directly calculating the gradient data using the gradient operator is usually of low precision, it has a certain impact on the precision and accuracy of the subsequently calculated Histogram of Oriented Gradient (HOG) features and the detection results. Therefore, in this embodiment, when calculating the gradient data of the current image frame to be processed, a separate algorithm is used to calculate the gradient data corresponding to each pixel of the current image frame according to the user's requirements. That is, a suitable algorithm is used to calculate the gradient magnitude and gradient angle corresponding to each pixel of the current image frame to obtain the gradient data of the current image frame, so as to improve the precision of the gradient magnitude and gradient angle values, that is, to improve the precision of the obtained gradient data.

[0035] Optionally, to reduce the computational load and improve the image quality of the current image frame during image processing, image preprocessing is performed on the current image frame before calculating the gradient data of the current image frame in the video sequence. Optionally, grayscale processing is performed on the current image frame, converting the multi-channel (e.g., RGB three-channel) current image frame into a single-channel grayscale current image frame, thereby reducing the size of the current image frame. Then, gamma correction is performed on the obtained current image frame to adjust the contrast of the current image frame, thereby reducing the impact of shadows and lighting changes in the current image frame, as well as suppressing interference from noise.

[0036] In this embodiment, since gradient data can reflect information such as the outline of an object in the current image frame, a suitable algorithm is used to calculate the gradient data of the current image frame, thereby improving the accuracy of the obtained gradient data. This allows the more accurate gradient data to better reflect information such as the outline of the object, thereby improving the accuracy of subsequent detection.

[0037] Step S102: Calculate the gradient direction histogram feature of the current image frame based on the gradient data to obtain the first feature.

[0038] Specifically, after determining the gradient data of the current image frame in the video sequence, the gradient orientation histogram (HHMA) feature of the current image frame is calculated based on the gradient data corresponding to each pixel. Specifically, when calculating the HHMA feature based on the gradient data of each pixel, the size of the cell unit is first set (i.e., the smallest statistical unit for the first feature extraction, typically a region containing multiple pixels, such as an 8*8 cell unit, meaning both length and width are 8 pixels). Then, the feature vector (i.e., the gradient orientation histogram) corresponding to the cell unit is statistically obtained based on the gradient data of each pixel corresponding to the cell unit. After obtaining the HHMA of each cell unit in the current image frame, several cell units (e.g., 3*3, which can be set by the user) are combined into a block. The feature vectors of all cell units within the block are concatenated to obtain the HHMA feature corresponding to this block. Finally, the HHMA features of all blocks in the current image frame are concatenated to form the final required HHMA feature of the current image frame, i.e., the first feature.

[0039] In this embodiment, since the gradient data is high-precision gradient data calculated using a suitable algorithm, the first feature of the current image frame of the video sequence is calculated based on the high-precision gradient data, making the first feature more accurate in describing the object, thereby improving the accuracy of subsequent detection.

[0040] Step S103: Determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, then perform a correction process on the first feature to obtain the corrected first feature.

[0041] The second feature mentioned above is the gradient orientation histogram feature of the previous image frame in the video sequence.

[0042] Specifically, since a video sequence is a continuous sequence of image frames, the changes between two consecutive image frames are usually small. However, changes in target motion or deformation may cause significant differences between two consecutive image frames, making it easy to miss the target when detecting later image frames. Therefore, in this embodiment, after calculating the first feature of the current image frame in the video sequence, the first feature (i.e., gradient orientation histogram) of the previous image frame in the video sequence is obtained and used as the second feature. The difference between the first feature and the second feature is calculated, and the degree of difference between the first feature and the second feature is determined. If the degree of difference between the first feature and the second feature is greater than a preset threshold (e.g., 0.2), it indicates that the current image frame in the video sequence has a large change relative to the previous image frame, which may affect subsequent detection. It is necessary to correct the first image corresponding to the current image frame to reduce the difference between the first feature and the second feature, so that subsequent detection is based on the corrected first feature, reducing the possibility of target loss (i.e., not detecting the target in the current image frame).

[0043] In this embodiment of the application, when the difference between the first feature and the second feature is greater than a preset threshold, the difference between the first feature and the second feature is reduced by correcting the first feature, thereby reducing the possibility of losing the target during subsequent detection and improving the target tracking effect.

[0044] Step S104: Detect the first feature to obtain a detection result. The detection result is used to indicate whether there is a detection target in the current image frame and the location information of the detection target when the detection target exists.

[0045] Specifically, after calculating the first feature of the current image frame of the video sequence, it detects whether a corresponding detection target (such as a pedestrian, the detection target is determined by the user according to the actual detection task) exists in the first feature, thereby obtaining the detection result corresponding to the current image frame. If a corresponding detection target is detected, information indicating that the detection target was detected is output, along with the location information of the detected target (i.e., the output detection result includes information indicating that a detection target was detected (e.g., the detection target exists in the current image frame) and the location information of the detection target (e.g., the location information of the four corner points of the rectangular region corresponding to the detection target)). If no detection target is detected in the first feature, i.e., no detection target is detected in the current image frame of the video sequence, information indicating that no detection target exists in the current image frame is output.

[0046] Optionally, since the changes in consecutive image frames of a video sequence are usually small in target tracking tasks, if the target is not detected in the current image frame, there is a possibility of false detection due to target deformation or other reasons causing the target to be lost in the current image frame. That is, the target may actually exist in the current image frame, but it is not detected. Therefore, when the detection result indicates that the target is not detected in the current image frame, the first feature of the current image frame can be extracted again for secondary detection. If the secondary detection result also indicates that the target is not detected in the current image frame, then it is determined that the target is not detected in the current image frame. If the secondary detection result indicates that the target is detected in the current image frame, then the previous detection result for the current image frame is determined to be a false detection, and the secondary detection result is taken as the detection result corresponding to the current image frame. Optionally, when the detection result indicates that the target is not detected in the current image frame, it is also possible to manually confirm whether the target exists in the current image frame (i.e., perform secondary detection by manual detection). This is not restricted here.

[0047] In this embodiment, since gradient histogram features can reflect the edge and corner information of objects in an image and are insensitive to changes in illumination, they can better reflect the shape of objects. Therefore, detection based on the obtained first feature, i.e., the gradient histogram feature of the current image frame of the video sequence, can better detect objects present in the current image frame, thereby accurately detecting existing targets and improving detection accuracy. Simultaneously, since the difference between the first feature and the second feature of the previous image frame is greater than a preset threshold, the first feature is corrected to reduce the difference between the first and second features. This reduces the impact of target motion and deformation on the current image frame in the video sequence, facilitating the detection and tracking of targets in the current image frame and improving target tracking efficiency.

[0048] In some embodiments, step S101 includes:

[0049] The gradient angle of each pixel in the current image frame of the video sequence is calculated using a coordinate rotation digital algorithm, thus obtaining the gradient angle of the current image frame.

[0050] The current image frame is convolved using a convolution algorithm to obtain the gradient magnitude corresponding to the current image frame.

[0051] The gradient data of the current image frame is determined based on the gradient angle and the gradient magnitude.

[0052] The aforementioned Coordinate Rotation Digital Computer (Cordic) algorithm replaces multiplication operations with basic addition, subtraction, and shift operations, gradually approximating the target value and thus obtaining the numerical solution of the function.

[0053] Specifically, since gradient data includes two data points: gradient angle and gradient magnitude, conventional gradient calculation directly uses gradient operators such as the Sobel operator to calculate the gradient of an image. Based on the gradient magnitude of each pixel with limited precision, the corresponding gradient angle is calculated, resulting in low precision of the gradient data. Therefore, in this embodiment, a coordinate rotation digital algorithm is used to continuously rotate and obtain the sine and cosine values ​​corresponding to each pixel. Then, the arctangent is calculated based on the obtained sine and cosine values ​​to obtain the angle (i.e., gradient angle) corresponding to each pixel. Convolution operation is then performed on the current image frame to obtain the gradient magnitude corresponding to each pixel. Thus, the gradient data of the current image frame is obtained based on the gradient angle and gradient magnitude corresponding to each pixel.

[0054] In this embodiment of the application, since the Cordic algorithm can obtain a function value with high accuracy through simple calculation, the use of the Cordic algorithm to calculate the gradient angle of the current image frame can improve the accuracy of the obtained gradient angle, thereby improving the accuracy of the obtained gradient data, so as to improve the detection accuracy when the first feature calculated based on the gradient data is subsequently detected.

[0055] In some embodiments, after step S102, the method further includes:

[0056] Gradient data is retrieved alternately from the first and second caches;

[0057] Before step S103 above, the following is also included:

[0058] The second feature is retrieved alternately from the first cache and the second cache.

[0059] Specifically, since data transmission requires waiting for data to be read before it can be processed, a first buffer and a second buffer acquire and store gradient data and second features to improve data transmission efficiency. The gradient data or second features stored in the first and second buffers are different. When gradient data or second features are needed for corresponding calculations, they are retrieved alternately from the first and second buffers. Specifically, within a specified time period, the first buffer transmits data (i.e., acquires gradient data or second features from the first buffer), and the second buffer reads data. Within this specified time period, the first buffer transmits all the stored data required for transmission, and the second buffer reads all the data required for storage. In the next specified time period, the first buffer reads data, and the second buffer transmits data (i.e., acquires gradient data or second features from the second buffer). Within this specified time period, the first buffer reads all the stored data required for transmission, and the second buffer transmits all the stored data required for transmission. In the process of calculating the gradient orientation histogram (HHMA) features of an image, the gradient data is usually directly calculated using gradient operators and transmitted in a FIFO (First Input First Output) manner so that the corresponding computing unit can calculate the HHMA features based on the gradient data without storing the gradient data. However, in this embodiment, a separate algorithm is used to calculate higher-precision gradient data, which needs to be stored for subsequent reuse. Furthermore, since setting up a separate data transmission path for gradient data requires corresponding modifications to the hardware, this embodiment reuses the first and second buffers used for transmitting the second feature to transmit gradient data. That is, gradient data or the second feature is alternately retrieved from the storage block using the first and second buffers, and then retrieved again from the first and second buffers, so that the first feature can be calculated based on the gradient data later. Alternatively, the gradient data of the current image frame and the gradient data corresponding to the second feature can be reused to calculate the difference between the first and second features.

[0060] Optionally, since gradient data includes gradient magnitude and gradient angle, when storing gradient data, the gradient magnitude and gradient angle corresponding to a pixel are concatenated to form the gradient data of a pixel for storage. However, when calculating the first feature based on the gradient data, it is necessary to calculate based on the specific gradient magnitude and gradient angle. Therefore, when retrieving gradient data from the first cache and the second cache, the gradient data in the first cache and the second cache are split to obtain the gradient magnitude and gradient angle corresponding to the gradient data. The gradient magnitude and gradient angle are then output to the calculation unit so that the calculation unit can calculate the first feature of the current image frame based on the gradient magnitude and gradient angle.

[0061] In this embodiment, gradient data and second features are transmitted by reusing the first and second buffers, eliminating the need to add a new transmission module to the hardware device to transmit gradient data, thus saving resources. At the same time, since the first and second buffers are used to transmit data alternately, data can be transmitted continuously during the data transmission process without waiting for the data to be read completely before transmission, thereby improving data transmission efficiency.

[0062] In some embodiments, retrieving gradient data from a first cache and a second cache, or retrieving a second feature from a first cache and a second cache, includes:

[0063] Step A: The first cache reads the first target data. At the same time, the second cache splits the read second target data and outputs the second target data. The first target data is the target data that the first cache needs to transmit in one transmission process, and the second target data is the target data that the second cache needs to transmit in one transmission process. The target data is gradient data or second feature.

[0064] Step B: If the first cache reads the first target data and the second cache outputs the second target data, the first cache splits the read first target data and outputs the first target data. At the same time, the second cache reads the new second target data.

[0065] Step C: If the first buffer outputs the first target data and the second buffer reads the new second target data, then return to step A and repeat steps A to C.

[0066] Specifically, since the data transmission unit can only process the data after reading it, the data transmission efficiency is low. Therefore, in this embodiment, two data transmission units, a first buffer and a second buffer, are used to transmit data alternately to obtain target data from the first buffer and the second buffer alternately. When the first buffer and the second buffer transmit the corresponding target data, one buffer is transmitting data and the other is reading data at the same time, thereby enabling the transmission of target data while reading data, reducing the time required to transmit target data. That is, in the actual transmission process, the first buffer and the second buffer read a fixed amount of target data each time (such as reading gradient data of 16 pixel positions at a time). During transmission, the first buffer reads the first target data, and at this time, the second buffer splits and transmits the second target data it has already read, so as to output the second target data to the corresponding module for subsequent processing. After the first buffer finishes reading the first target data and the second buffer finishes transmitting the second target data, the buffer for reading and transmitting data is replaced. That is, the first buffer is controlled to transmit the first target data it has read, and at the same time, the second buffer is controlled to read the new second target data (that is, the second buffer starts a new transmission process and transmits other target data that is still waiting to be transmitted). After the first buffer finishes transmitting the first target data and the second buffer finishes reading the new second target data, the first buffer also begins a new transmission, that is, reading the new first target data. At the same time, the second buffer transmits the new second target data it has read. This process continues, with the first and second buffers alternating between reading and transmitting target data. This ensures that data is being transmitted at every moment during the data transmission process, without having to wait for the data to be read before transmitting, thereby improving the efficiency of data transmission.

[0067] For example, suppose the target data to be transmitted is the gradient data of the current image frame of a video sequence. This gradient data includes gradient data from 64 pixels. During a single transmission, the first and second buffers read and transmit gradient data from 6 pixels. Then, when transmitting this gradient data...

[0068] During the first transmission, the first buffer reads the gradient data (i.e., the first target data) of 6 pixels with pixel positions from 0 to 5. Since the second buffer has not yet read any data during the first transmission, only the first buffer reads the first target data during the first transmission, and no data is transmitted. After the first buffer finishes reading the first target data, the first transmission ends.

[0069] During the second transmission, the first buffer splits the read first target data and outputs the split first target data to the corresponding calculation unit. At the same time, while the first buffer is transmitting the first target data, the second buffer reads the gradient data (second target data) of 6 pixels at pixel positions 6 to 11. After the first buffer unit has finished transmitting the first target data and the second buffer has finished reading the second target data, the second transmission ends.

[0070] During the third transmission, the first buffer reads the gradient data (new first target data) of 6 pixels at pixel positions 12 to 17. At the same time, the second buffer splits the read second target data and outputs the split second target data to the corresponding calculation unit. After the first buffer unit has finished reading the new first target data and the second buffer has finished transmitting the second target data, the third transmission ends.

[0071] During the fourth transmission, the first buffer splits the newly read first target data and outputs the split first target data to the corresponding calculation unit. Simultaneously, while the first buffer is transmitting the first target data, the second buffer reads the gradient data (new second target data) of six pixels at pixel positions 18 to 23. The fourth transmission ends after both the first and second buffers have finished transmitting the new first target data. This process continues, with the first and second buffers alternating between reading and transmitting until the gradient data (i.e., target data) of the current image frame has been completely transmitted.

[0072] In this embodiment of the application, by alternately reading and transmitting data through the first and second caches, the transmission module can continuously transmit data during the data transmission process without having to wait for data to be read, thereby reducing the time required to transmit the target data, improving data transmission efficiency, and thus improving image processing efficiency.

[0073] In some embodiments, step S103, when determining the degree of difference between the first feature and the second feature, includes:

[0074] Based on the gradient data corresponding to the first feature and the gradient data corresponding to the second feature, the gradient difference between the first feature and the second feature is calculated, and the degree of difference between the first feature and the second feature is determined based on the gradient difference.

[0075] Specifically, since gradients can reflect information such as the outline of objects in an image, and in a video sequence, the changes in objects between two consecutive image frames are usually very small, that is, the data corresponding to the same object are basically the same, in this embodiment of the application, the gradient difference between the current image frame and the previous image frame in the video sequence is calculated based on the gradient data corresponding to the first feature and the gradient data corresponding to the second feature. The degree of difference between the first feature and the second feature (i.e., the degree of difference between the current image frame and the previous image frame in the video sequence) is determined based on the gradient difference, thereby determining the changes of each object in the current image frame relative to the objects in the previous image frame, so as to correct the current image frame when the changes in the current image frame are large.

[0076] Optionally, since the gradient data of the same object are basically the same, and the objects in two consecutive image frames of a video sequence are usually the same and do not change much, when calculating the gradient difference between the first feature and the second feature based on the gradient data, the gradient difference is calculated based on the pixels corresponding to the gradient data of the same object (for example, objects in a region of the same size in two consecutive image frames whose gradient data of each pixel is basically the same are considered as the same object). This determines the difference between the object in the current image frame and the corresponding object in the previous image frame, thus identifying which objects have changed significantly. Optionally, the target tracking task usually provides the size and position of the target in the initial frame of the video sequence. When calculating the gradient difference between the first feature and the second feature, the gradient difference of each pixel in the region corresponding to the position of the target in the initial frame given by the target tracking task can also be directly calculated.

[0077] Optionally, since the storage block stores the gradient data of the current image frame, when calculating the gradient difference based on the gradient data corresponding to the first feature and the gradient data corresponding to the second feature, the corresponding gradient data can be directly obtained from the storage through the transmission module, and the stored gradient data can be reused to calculate the gradient difference. There is no need to calculate the gradient data again, thereby reducing the amount of computation and the computation time in the image processing process.

[0078] In this embodiment of the application, since gradient data can reflect the contour information of an object, the degree of difference between the first feature and the second feature is determined based on the gradient difference determined by the gradient data. This is equivalent to determining the difference between the objects in the current image frame and the previous image frame, so that when the changes in the objects in the current image frame are large, the current image frame can be corrected to reduce the changes caused by target movement or deformation.

[0079] In some embodiments, step S103, when modifying the first feature to obtain the modified first feature, includes:

[0080] Based on the difference between the first feature and the second feature, the first feature is interpolated to obtain the corrected first feature.

[0081] Specifically, since gradient data can reflect the contour information of an object, and the gradient data of the same object are basically the same, the gradient difference between the first feature and the second feature can reflect the difference between the object in the current image frame and the corresponding object in the previous image frame, that is, the change of the object in the current image frame compared with the object in the previous image frame. Therefore, the region where the first feature differs from the second feature is determined based on the gradient difference, that is, the existing gradient difference. Based on the existing gradient difference, interpolation processing (such as nearest neighbor interpolation) is performed on the region corresponding to the first feature (i.e. the pixels with gradient difference) to reduce the change of the first feature relative to the second feature.

[0082] In some embodiments, the above image processing method further includes:

[0083] Based on the target region in the current image frame, it is determined whether the first feature needs to be aligned. The alignment process is used to convert the format of the first feature through an affine transformation.

[0084] If the first feature needs to be aligned, then an affine transformation is performed on the first feature to obtain the aligned first feature.

[0085] Specifically, since targets in target tracking tasks are usually moving, the targets in the image frames of the video sequence may be deformed, causing the target to be undetectable. Therefore, in order to reduce false detections caused by target deformation, in this embodiment, before detecting the target present in the first feature, it is determined whether the target region is a non-standard shape (such as a rectangle) based on the target region in the current image frame (such as the region corresponding to each object reflected by the first feature). If the target region is a non-standard shape, it is determined that the first feature needs to be aligned. At this time, the target position corresponding to each corner point after converting the target region into a standard shape target region is determined based on each corner point of the target region, and the target position corresponding to each feature point (such as each pixel or each cell unit) in the target region after converting it into a standard shape target region is determined. Thus, the first feature is transformed and mapped according to the determined target positions to obtain the aligned first feature after affine transformation.

[0086] In this embodiment, since the first feature is subjected to affine transformation when alignment is required, the features of the deformed object can be converted into the features of an object with a standard shape, thereby reducing the impact of deformation on subsequent detection and improving the accuracy of target tracking.

[0087] In some embodiments, when the image processing of the current image frame includes Fast Fourier Transform (FFT) processing, the discontinuities in the image spatial domain (such as the unevenness of the edges in the image) can cause the spectrum obtained after the Fast Fourier Transform to be chaotic and complex. Therefore, in order to reduce spectral leakage and avoid obtaining a chaotic spectrum, Hanning windowing is applied to the first feature after it is calculated, or Hanning windowing is applied to the original current image frame to reduce the discontinuities in the current image frame, so that the spectrum with concentrated energy is obtained after subsequent FFT processing.

[0088] The following examples illustrate the overall flow of the image processing method provided in some embodiments of this application.

[0089] For example, suppose we need to perform a pedestrian tracking task, that is, to track and detect target pedestrians in a video sequence. Given a video sequence and the target size and position information in the initial frame (i.e., the first image frame) of the video sequence, the processing procedure for each image frame when detecting and tracking the target pedestrian is as follows:

[0090] The Cordic algorithm is used to calculate the gradient angle corresponding to each pixel of the current image frame in the video sequence to obtain a high-precision gradient angle. The gradient magnitude corresponding to each pixel of the current image frame is calculated by convolution. Thus, the gradient data of the current image frame is obtained based on the gradient angle and gradient magnitude of each pixel, and the gradient data of the current image frame is stored in the storage block.

[0091] Gradient data is transmitted to the corresponding computation unit via a transmission module comprising a first buffer and a second buffer. The computation unit calculates the gradient direction histogram based on the gradient data of the current image frame to obtain the first feature. Since pedestrians are typically moving, there may be issues with facial asymmetry or shape changes in the pedestrians in different image frames of the video sequence, leading to the failure to detect the target pedestrian. Therefore, during image processing, it is determined whether the target region in the current image frame is a standard-shaped region, thereby determining whether alignment processing of the first feature is necessary. When alignment processing of the first feature is required, an affine transformation is performed on the first feature to achieve alignment, resulting in the aligned first feature. Specifically, the first and second buffers of the transmission module alternately transmit data when transmitting gradient data or the second feature.

[0092] The second feature is transmitted to the corresponding computing unit through the first and second buffers. The gradient difference between the first and second features is calculated, and the degree of difference between the first and second features is calculated based on the gradient difference. Since the degree of difference is greater than a preset threshold, it indicates that the change of the current image frame relative to the previous image frame is too large. Therefore, it is necessary to perform interpolation processing on the first feature corresponding to the current image frame to obtain the corrected first feature, thereby realizing the correction of the current image frame, reducing the change of the current image frame, and reducing the possibility of losing the target pedestrian.

[0093] After obtaining the corrected first feature, the first feature is detected to determine whether there is a target pedestrian in the current image frame. If a target pedestrian is detected in the current image frame, the detection result including the detected target pedestrian and the location information of the detected target pedestrian is output, thereby realizing the detection of the current image frame.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] Example 2:

[0096] Corresponding to the image processing method described in the above embodiments, Figure 2 A structural block diagram of an image processing apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0097] Reference Figure 2 The device includes: a gradient calculation module 21, a first feature acquisition module 22, a correction module 23, and a detection module 24. Among them,

[0098] Gradient calculation module 21 is used to determine the gradient data of the current image frame of the video sequence;

[0099] The first feature acquisition module 22 is used to calculate the gradient direction histogram feature of the current image frame based on the gradient data to obtain the first feature;

[0100] The correction module 23 is used to determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, the first feature is corrected to obtain the corrected first feature. The second feature is the gradient direction histogram feature of the previous frame of the current image frame in the video sequence.

[0101] Detection module 24 is used to detect the first feature and obtain a detection result. The detection result is used to indicate whether there is a detection target in the current image frame and the location information of the detection target when the detection target exists.

[0102] In this embodiment, since gradient histogram features can reflect the edge and corner information of objects in an image and are insensitive to changes in illumination, they can better reflect the shape of objects. Therefore, detection based on the obtained first feature, i.e., the gradient histogram feature of the current image frame of the video sequence, can better detect objects present in the current image frame, thereby accurately detecting existing targets and improving detection accuracy. Simultaneously, since the difference between the first feature and the second feature of the previous image frame is greater than a preset threshold, the first feature is corrected to reduce the difference between the first and second features. This reduces the impact of target motion and deformation on the current image frame in the video sequence, facilitating the detection and tracking of targets in the current image frame and improving target tracking efficiency.

[0103] In some embodiments, the gradient calculation module 21 includes:

[0104] An angle calculation unit is used to calculate the gradient angle of each pixel of the current image frame of the video sequence using a coordinate rotation digital algorithm, so as to obtain the gradient angle of the current image frame.

[0105] The amplitude calculation unit is used to perform convolution operation on the current image frame using a convolution algorithm to obtain the gradient amplitude corresponding to the current image frame.

[0106] The gradient acquisition unit is used to determine the gradient data of the current image frame based on the gradient angle and the gradient magnitude.

[0107] In some embodiments, the image processing apparatus further includes:

[0108] The gradient data acquisition module is used to alternately acquire gradient data from the first cache and the second cache.

[0109] The second feature acquisition module acquires the second feature alternately from the first cache and the second cache.

[0110] In some embodiments, the image processing apparatus further includes:

[0111] The first transmission module is used to read the first target data from the first buffer. At the same time, the second buffer splits the read second target data and outputs the second target data. The first target data is the target data that the first buffer needs to transmit in one transmission process, and the second target data is the target data that the second buffer needs to transmit in one transmission process. The target data is gradient data or a second feature.

[0112] The second transmission module is configured to, if the first cache reads the first target data and the second cache outputs the second target data, then the first cache splits the read first target data and outputs the first target data, while the second cache reads the new second target data.

[0113] The loop transmission module is configured to return to step A and repeat steps A to C if the first buffer outputs the first target data and the second buffer has finished reading the new second target data.

[0114] In some embodiments, the correction module 23 includes:

[0115] The difference degree calculation unit is used to calculate the gradient difference between the first feature and the second feature based on the gradient data corresponding to the first feature and the gradient data corresponding to the second feature, and to determine the difference degree between the first feature and the second feature based on the gradient difference.

[0116] In some embodiments, the correction module 23 further includes:

[0117] The correction unit is used to perform interpolation processing on the first feature based on the difference between the first feature and the second feature to obtain the corrected first feature.

[0118] In some embodiments, the image processing apparatus further includes:

[0119] The transformation confirmation module is used to determine whether the first feature needs to be aligned based on the target region in the current image frame. The alignment process is used to convert the format of the first feature through affine transformation.

[0120] The alignment module is used to perform an affine transformation on the first feature if the first feature needs to be aligned, so as to obtain the aligned first feature.

[0121] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0122] Example 3:

[0123] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 3 As shown, the terminal device 3 in this embodiment includes: at least one processor 30 ( Figure 3The diagram shows only one processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above method embodiments.

[0124] The terminal device 3 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of terminal device 3 and does not constitute a limitation on terminal device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0125] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0126] In some embodiments, the memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. In other embodiments, the memory 31 may be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 may include both internal and external storage units of the terminal device 3. The memory 31 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0128] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0129] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0130] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Determine the gradient data of the current image frame in the video sequence; The gradient direction histogram feature of the current image frame is calculated based on the gradient data to obtain the first feature; Determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, then the first feature is corrected to obtain the corrected first feature, so that subsequent detection is based on the corrected first feature. The second feature is the gradient orientation histogram feature of the previous image frame of the current image frame in the video sequence. The first feature is detected to obtain a detection result, which is used to indicate whether the current image frame has a detection target and, if the detection target exists, to indicate the location information of the detection target. If the detection result indicates that the target is not present in the current image frame, the first feature of the current image frame is extracted again, and the first feature is subjected to secondary detection to obtain the final detection result. The step of modifying the first feature to obtain the modified first feature includes: Based on the gradient difference between the first feature and the second feature, the pixels in the first feature that have the gradient difference are interpolated to obtain the corrected first feature. The gradient difference is used to reflect the difference between each object in the current image frame and the corresponding object in the previous image frame.

2. The image processing method as described in claim 1, characterized in that, Before calculating the gradient orientation histogram features of the current image frame based on the gradient data, the method further includes: The gradient data is retrieved alternately from the first cache and the second cache; Before determining the degree of difference between the first feature and the second feature, the method further includes: The second feature is retrieved alternately from the first cache and the second cache.

3. The image processing method as described in claim 1, characterized in that, The gradient data for determining the current image frame of the video sequence includes: The gradient angle of each pixel in the current image frame of the video sequence is calculated using a coordinate rotation digital algorithm to obtain the gradient angle of the current image frame; The current image frame is convolved using a convolution algorithm to obtain the gradient magnitude corresponding to the current image frame; The gradient data of the current image frame is determined based on the gradient angle and the gradient magnitude.

4. The image processing method as described in claim 1, characterized in that, The method further includes: Whether the first feature needs to be aligned is determined based on the target region in the current image frame. The alignment process is used to convert the format of the first feature through an affine transformation. If the first feature needs to be aligned, then an affine transformation is performed on the first feature to obtain the aligned first feature.

5. The image processing method according to any one of claims 1 to 4, characterized in that, Determining the degree of difference between the first feature and the second feature further includes: Based on the gradient data corresponding to the first feature and the gradient data corresponding to the second feature, the gradient difference between the first feature and the second feature is calculated, and the degree of difference between the first feature and the second feature is determined based on the gradient difference.

6. An image processing apparatus, characterized in that, include: The gradient calculation module is used to determine the gradient data of the current image frame in the video sequence; The first feature acquisition module is used to calculate the gradient direction histogram feature of the current image frame based on the gradient data to obtain the first feature; The correction module is used to determine the degree of difference between the first feature and the second feature. If the degree of difference is greater than a preset threshold, the first feature is corrected to obtain the corrected first feature, so that subsequent detection is based on the corrected first feature. The second feature is the gradient orientation histogram feature of the previous image frame of the current image frame in the video sequence. The detection module is used to detect the first feature and obtain a detection result. The detection result is used to indicate whether there is a detection target in the current image frame and the location information of the detection target when the detection target exists. If the detection result indicates that the target is not present in the current image frame, the first feature of the current image frame is extracted again, and the first feature is subjected to secondary detection to obtain the final detection result. The correction module includes: The correction unit is used to perform interpolation processing on the pixels in the first feature that have the gradient difference based on the gradient difference between the first feature and the second feature, so as to obtain the corrected first feature. The gradient difference is used to reflect the difference between each object in the current image frame and the corresponding object in the previous image frame.

7. The image processing apparatus as claimed in claim 6, characterized in that, Also includes: A gradient data acquisition module is used to alternately acquire the gradient data from a first cache and a second cache; The second feature acquisition module is used to alternately acquire the second feature from the first cache and the second cache.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images