Image processing method, image processing device, computer-readable storage medium, electronic device, and computer program

Abstract

The embodiments of the present invention disclose an image processing method and its device, electronic device, storage medium, computer program, and computer program product, which includes: acquiring at least two frames of road images through an image acquisition component provided in a traveling device, determining posture change information of the traveling device between two frames of the road images of the at least two frames using a phase-only correlation method, and determining posture information of the traveling device based on the posture change information and reference posture information of the traveling device.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is based on a Chinese patent application filed on March 24, 2022, bearing application number 202210303559.6 and entitled "Image processing method and apparatus therefor, electronic device, and storage medium," and claims priority to that Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer vision technology, and in particular to an image processing method and apparatus therefor, an electronic device, a storage medium, a computer program, and a computer program product. [Background technology]

[0003] An accurate and robust calibration system is crucial for autonomous driving systems. The calibration system determines the reference coordinate system for the measurements output by each sensor, improving accuracy and consistency while the vehicle is moving. However, road irregularities and vehicle acceleration / deceleration cause changes in the vehicle's attitude relative to the ground.

[0004] This problem is usually solved in two technical aspects: software and hardware. In the hardware aspect, high-precision inertial navigation positioning sensors are usually used to provide accurate position and orientation information, but this method is usually relatively expensive. In the software aspect, feature point matching methods are usually used to calculate position and orientation changes, but this method is relatively time-consuming. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the existing technical problems, embodiments of the present invention provide an image processing method and apparatus, an electronic device, a storage medium, a computer program, and a computer program product. [Means for solving the problem]

[0006] To achieve the above objectives, the technical solutions of the embodiments of the present invention are realized as follows:

[0007] An embodiment of the present invention provides an image processing method, the method comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining, using a phase-only correlation method, attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images; Determining attitude information of the traveling device based on the attitude change information and reference attitude information of the traveling device.

[0008] An embodiment of the present invention further provides an image processing apparatus, the apparatus comprising: an acquisition unit; an attitude offset sensing unit; and an attitude determination unit, wherein: The acquisition unit is configured to acquire at least two frames of road images via an image acquisition component provided on a traveling device; The attitude offset sensing unit is configured to determine attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images using a phase-only correlation method; The attitude determination unit is configured to determine attitude information of the traveling equipment based on the attitude change information and reference attitude information of the traveling equipment.

[0009] An embodiment of the present invention further provides a computer-readable storage medium having stored thereon a computer program for causing a processor to perform the steps of the method according to an embodiment of the present invention.

[0010] An embodiment of the present invention further provides an electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the processor executing the program to implement the steps of the method according to the embodiment of the present invention.

[0011] An embodiment of the present invention provides a computer program comprising computer-readable code, which, when read and executed by a computer, causes the computer to perform some or all of the steps of the method in any one of the embodiments of the present application.

[0012] An embodiment of the present invention provides a computer program product, the computer program product including a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program causing the computer to perform some or all of the steps of the method in any one of the embodiments of the present application when read and executed by a computer. [Effects of the Invention]

[0013] An embodiment of the present invention provides an image processing method and its apparatus, electronic device, storage medium, computer program, and computer program product. The method includes: acquiring at least two frames of road images via an image acquisition component provided in a mobile device; determining posture change information of the mobile device between two of the at least two frames of road images using a phase-only correlation method; and determining posture information of the mobile device based on the posture change information and reference posture information of the mobile device. According to a technical solution of an embodiment of the present invention, the posture change information between the road images is determined using a phase-only correlation method. Compared to a method using a high-precision inertial navigation positioning sensor, the embodiment of the present invention can obtain accurate position and posture information while reducing costs. Compared to a method using feature point matching to calculate position and posture information, the embodiment of the present invention can reduce the amount of calculation and is applicable to a monocular visual sensor. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is an exemplary flowchart 1 of an image processing method according to an embodiment of the present invention. [Figure 2] FIG. 10 is a schematic diagram of a process for processing phase offset information in the image processing method according to the embodiment of the present invention. [Figure 3] 4 is a schematic diagram of phase offset information in the image processing method according to the embodiment of the present invention; [Figure 4] 1 is an exemplary structural diagram 1 of a configuration of an image processing device according to an embodiment of the present invention. [Figure 5] 2 is an exemplary structural diagram 2 of the configuration of the image processing device according to the embodiment of the present invention. [Figure 6] 1 is an exemplary structural diagram of a hardware configuration of an electronic device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0015] In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings necessary for the embodiments have been briefly introduced above. The drawings herein are incorporated into and constitute a part of this specification, and these drawings are for illustrating the embodiments consistent with the present application and are used together with this specification to explain the technical solutions of the embodiments of the present invention. The drawings only illustrate some of the embodiments of the present invention, and therefore should not be considered as limiting the scope of the present application. It should be understood that those skilled in the art can also obtain other related drawings according to these drawings without creative efforts.

[0016] In the following, the present application will be explained in more detail with reference to the drawings and specific embodiments.

[0017] In embodiments of the present invention, the terms "comprise," "include," or any other variations thereof are intended to cover a non-exclusive inclusiveness, whereby a method or apparatus comprising a series of elements not only includes the elements explicitly described, but also includes other elements not explicitly listed, or includes elements inherent in implementing the method or apparatus. Unless otherwise limited, an element defined by the phrase "comprises" does not exclude the presence of another related element (e.g., a step in a method or a unit in an apparatus, where an example unit may be a partial circuit, a partial processor, a partial program or software, etc.) in the method or apparatus that includes the element.

[0018] For example, although an image processing method according to an embodiment of the present invention may include a series of steps, the image processing method according to an embodiment of the present invention is not limited to the steps described. Similarly, although an image processing device according to an embodiment of the present invention includes a series of modules, the device according to an embodiment of the present invention is not limited to including the explicitly described modules and may further include a module for obtaining related information or a module for processing based on the information.

[0019] The term "and / or" used herein describes an association relationship between associated objects and indicates that three relationships may exist; for example, A and / or B can represent three cases: A exists independently, both A and B exist, and B exists independently. Furthermore, the term "at least one" used herein indicates any one of a plurality of elements, or any combination of at least two of a plurality of elements; for example, including at least one of A, B, and C indicates including any one or more elements selected from the set consisting of A, B, and C.

[0020] An embodiment of the present invention provides an image processing method. Figure 1 is an exemplary flowchart 1 of the image processing method of an embodiment of the present invention. As shown in Figure 1, the method includes the following steps:

[0021] In step S101, at least two frames of road images are acquired via an image acquisition component provided in the traveling equipment.

[0022] In step S102, a phase-only correlation method is used to determine information on a change in attitude of the traveling equipment between two frames of road images among the at least two frames of road images.

[0023] In step S103, attitude information of the traveling device is determined based on the attitude change information and the reference attitude information of the traveling device.

[0024] In some embodiments of the present invention, the mobile device may be an autonomous vehicle, a vehicle equipped with an advanced driver assistance system, a robot, etc. The image processing method of the embodiments of the present invention is applied to an electronic device, and the electronic device may be an in-vehicle device, a cloud platform, or other computer device. In some embodiments, the in-vehicle device may be a thin client, a thick client, a microprocessor-based system, a small computer system, etc. implemented in the vehicle, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system, etc. In some possible embodiments, the image processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0025] In some embodiments of the present invention, the on-vehicle device may be communicatively connected to a vehicle sensor, a positioning device, etc., and the on-vehicle device may obtain data collected by the vehicle sensor and geographical location information reported by the positioning device, etc. via the communication connection. In some embodiments, the vehicle sensor may be at least one of a millimeter-wave radar, a laser radar (LIDAR), a camera, etc., and the positioning device may be a device that provides positioning services based on at least one positioning system selected from the group consisting of a Global Positioning System (GPS), a Beidou satellite navigation system, and a Galileo satellite navigation system.

[0026] In one example, the in-vehicle device may be an Advanced Driving Assistant System (ADAS), which is installed in a vehicle and can acquire real-time position information of the vehicle from a positioning device of the vehicle, and can also acquire image data, radar data, etc. representing vehicle surrounding environment information from sensors of the vehicle. Here, optionally, the ADAS can transmit vehicle driving data including the real-time position information of the vehicle to a cloud platform, and in this way, the cloud platform can receive at least one of the real-time position information of the vehicle, the image data representing vehicle surrounding environment information, and the radar data.

[0027] In an embodiment of the present invention, road images are acquired via an image collection component (i.e., the above-mentioned sensor such as a camera) provided on the traveling equipment, and the image collection component collects road images or environmental images around the vehicle in real time as the traveling equipment moves.

[0028] According to the technical solution of the embodiment of the present invention, a phase-only correlation method is used to determine pose change information between road images. Compared with the method of adopting a high-precision inertial navigation positioning sensor, the embodiment of the present invention can reduce costs while obtaining accurate position and orientation information. Compared with the method of adopting a feature point matching method to calculate position and orientation information, the embodiment of the present invention can reduce the amount of calculation and is applicable to monocular visual sensors.

[0029] In some alternative embodiments of the present invention, using the phase-only correlation method to determine posture change information of the moving equipment between two frames of road images of the at least two frames of road images includes determining phase offset information between the two frames of road images based on a region of interest within the two frames of road images, and determining posture change information of the moving equipment between the two frames of road images based on the phase offset information.

[0030] In the field of image processing, a Region of Interest (ROI) is a region selected from an image (i.e., a road image in an embodiment of the present invention), which is a key point of interest for image analysis or processing. In some embodiments, the ROI can be determined in the form of a square, a circle, an ellipse, an irregular polygon, etc. In an embodiment of the present invention, a square is used as an example to determine the ROI in the road image.

[0031] In an embodiment of the present invention, by setting a region of interest (ROI) in a road image, phase offset information between road images is determined using a phase-only correlation method based on the ROI in each road image, and then, based on the phase offset information, posture change information of the traveling equipment between the road images of two frames is determined.Compared to a method of calculating position and posture information by adopting a feature point matching method, this embodiment of the present invention can significantly reduce the amount of calculation.

[0032] In some alternative embodiments, the region of interest in the road image of each frame is determined by the following method: determining one or at least two regions of interest in the road image of each frame, and at least one of the one or at least two regions of interest has at least one of the following characteristics: overlapping with the horizon line; and including a vanishing point.

[0033] In an embodiment of the present invention, one or at least two regions of interest are set in a road image of each frame, where at least one region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point. In some embodiments, when a road image includes one region of interest, the region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point. When a road image includes at least two regions of interest, each region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point.

[0034] Optionally, each region of interest of the at least two regions of interest may overlap with at least one other region of interest.

[0035] Here, the size and aspect ratio of the set region of interest can be set according to actual needs, and the embodiment of the present invention is not limited thereto.

[0036] In a typical situation, the road image is acquired through an image acquisition component provided on a mobile device, and the mobile device generally travels on a road, i.e., the road image generally includes lanes on the road, and the vanishing points represent parallel lines on the actual road, such as points where lanes or road edge lines visually intersect in the road image. The region of interest in an embodiment of the present invention includes the vanishing points.

[0037] In some alternative embodiments, the at least one region of interest is symmetrical with respect to the horizon.

[0038] In some alternative embodiments, the pixel points on each side of the at least one region of interest are 2 n , where n is a positive integer.

[0039] To facilitate image processing, in an embodiment of the present invention, the size of the at least one region of interest is a size that can be easily processed by image processing, for example, when the image processing process includes a Fast Fourier Transform (FFT), the pixel points on each side of the at least one region of interest are 2 to the power of n, where n is a positive integer.

[0040] In some alternative embodiments, there are no moving objects in the region of interest, or the proportion of the moving objects occupying the region of interest is less than or equal to a first threshold.

[0041] In some embodiments of the present invention, the selected region of interest may not include any moving objects, or the proportion of the moving objects in the selected region of interest that occupy the region of interest may be small, e.g., the proportion of the moving objects occupying the region of interest may be equal to or less than a first threshold. As a result, the calculated phase offset information between two frames of road images is not affected by the moving objects or is only slightly affected by the moving objects, and the acquired phase offset information and attitude offset information of the moving device may be more accurate. Here, the value of the first threshold may be set according to actual conditions, and the embodiments of the present invention are not limited thereto. In some embodiments, the moving objects may be other moving devices (e.g., other vehicles) in the road images.

[0042] In some alternative embodiments, when the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest, wherein the first region of interest includes the vanishing point.

[0043] In another embodiment, the first region of interest is also called a main region of interest (Main ROI), and the second region of interest is also called an auxiliary region of interest (Auxiliary ROI). In a different scenario, the electronic device can determine phase offset information between the road images of the two frames based on at least one of the first region of interest and the second region of interest in the road images of the two frames.

[0044] Optionally, a size of the second region of interest is smaller than a size of the first region of interest.

[0045] Optionally, the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps with the first region of interest.

[0046] In some alternative embodiments of the present invention, determining phase offset information between the road images of the two frames based on an area of ​​interest within the road images of the two frames includes determining phase offset information between the road images of the two frames based on an area of ​​interest within the road image of each of the road images of the two frames.

[0047] In an embodiment of the present invention, if there are no moving objects in the region of interest, or if the proportion of moving objects occupying the region of interest is below a first threshold, that is, if the phase offset information between the road images of two frames is not affected by moving objects or is only slightly affected by moving objects, the phase offset information between the road images of the two frames can be calculated based on the region of interest in the road images of each frame.

[0048] In some alternative embodiments of the present invention, a moving object is present in the first region of interest, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes determining phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames.

[0049] In an embodiment of the present invention, if a moving object is present in a first region of interest (e.g., a main region of interest), it is highly likely that no moving object is present in the second region of interest because the second region of interest and the first region of interest are set at different positions, and phase offset information between the road images of the two frames is determined based on a second region of interest (e.g., an auxiliary region of interest) in the road images of the two frames. In another embodiment, if no moving object is present in the first region of interest (e.g., a main region of interest), the phase offset information between the road images of the two frames can be determined based only on the first region of interest (e.g., the main region of interest), or the phase offset information between the road images of the two frames can be determined based on the first region of interest and the second region of interest.

[0050] In some alternative embodiments, if the region of interest in the first road image contains a moving object, or if the proportion of moving objects occupying the region of interest in the first road image is greater than or equal to a second threshold, the first road image is not used to determine phase offset information between the first road image and other road images, i.e., the first road image is not used to determine phase offset information between the first road image and other road images, i.e., the first road image is not used to calculate the phase offset information, thereby excluding the influence of moving objects in the region of interest of the road image on the phase offset information obtained by the calculation.

[0051] In some alternative embodiments, a moving object is present in a region of interest in the road image of each of the two frames, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes determining first phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames, detecting the proportion of the region where the moving object is located in each of the regions of interest in the road images of the two frames, and determining phase offset information between the road images of the two frames based on the proportion and the first phase offset information.

[0052] In an embodiment of the present invention, when a moving object is present in both of the regions of interest in road images of two frames for determining phase offset information, the calculated phase offset information between the road images of the two frames (referred to as first phase offset information here) is calibrated to calibrate the influence of the moving object on the calculated phase offset information. For example, after calculating the first phase offset information based on the regions of interest in the road images of the two frames, the electronic device can detect the proportion of the moving object within the region of interest, for example, detect a detection frame of the moving object to determine an overlapping region between the moving object and the region of interest (if the moving object is large, it is possible that the entire moving object is not within the region of interest), and further determine the proportion of the size (or area) of the overlapping region to the region of interest, and determine the phase offset information between the road images of the two frames based on the proportion and the first phase offset information.

[0053] Optionally, the ratio may be multiplied by the first phase offset information to obtain phase offset information between the road images of the two frames.

[0054] In some alternative embodiments, the road image of each frame includes at least two regions of interest, and determining phase offset information between the road images of the two frames based on the regions of interest in the road images of the two frames includes determining second phase offset information between the road images of the two frames based on each corresponding group of regions of interest in the road images of the two frames, wherein the positions of each region of interest in each group in each road image correspond to each other; and selecting one second phase offset information from the at least two second phase offset information as the phase offset information between the road images of the two frames, or determining a median phase change information among the at least two second phase offset information and determining the median phase change information as the phase offset information between the road images of the two frames, or determining an average value of the at least two second phase offset information and determining the average value as the phase offset information between the road images of the two frames.

[0055] In an embodiment of the present invention, if a road image includes at least two regions of interest, for example, road image 1 includes three regions of interest and road image 2 includes three corresponding regions of interest. Each region of interest in road image 1 can form a group of regions of interest with a corresponding region of interest in road image 2, and the positions of the regions of interest in each group in the respective road images correspond to each other. Then, second phase offset information between the road images of two frames can be calculated for each group of regions of interest. In one embodiment, one second phase offset information is selected from the at least two sets of second phase offset information and used as the phase offset information between the road images of the two frames in subsequent calculations of posture information of the traveling device. In a second embodiment, median phase offset information among the at least two sets of second phase offset information can be determined, and the median phase change information can be determined as the phase offset information between the road images of the two frames. In the case of the median phase offset information, the at least two second phase offset information are sorted in descending or ascending order, and the second phase offset information located in the middle is the median phase offset information. As a third embodiment, at least two pieces of second phase offset information are added together to find an average value, and the average value obtained by calculation can be used as the phase offset information between the road images of two frames.

[0056] In the third embodiment, before adding at least two pieces of second phase offset information to calculate an average value, the method further includes removing abnormal second phase offset information from the at least two pieces of second phase offset information, and adding the removed second phase offset information to calculate an average value to obtain phase offset information between road images of two frames. Generally, the second phase offset information calculated based on each region of interest should be similar, i.e., the difference between each piece of second phase offset information does not exceed a certain threshold. The abnormal value may be second phase offset information whose difference with any second phase offset information exceeds the threshold. That is, second phase offset information with a large difference is considered an abnormal value and needs to be removed.

[0057] In some alternative embodiments of the present invention, determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames includes: extracting sub-images corresponding to regions of interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image; performing grayscale processing on the first sub-image and the second sub-image, respectively, to obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image; performing Fourier transform processing on the first grayscale image and the second grayscale image, respectively, and performing normalized cross processing on corresponding pixel points in the processed first grayscale image and the processed second grayscale image to obtain processed images; performing inverse Fourier transform processing on the processed images, to determine peak positions based on values ​​of each pixel point in the processed images, and determining phase offset information between the road images of the two frames based on the peak positions.

[0058] Here, the normalized cross-correlation process refers to a process of performing a normalization process on each pixel of the first and second grayscale images that have been Fourier transformed, and then calculating the cross power spectrum for each pixel of the two normalized images.Furthermore, the processed images are subjected to an inverse Fourier transform process, which is also called a normalized cross-correlation process.

[0059] In some embodiments, referring to Figures 2 and 3, taking road images of two frames as an example, assuming that the road images of each frame each have one region of interest, for example, the rectangular framed region in image 1 is the region of interest of image 1 (201), and the rectangular framed region in image 2 (202) is the region of interest of image 2, and since the positions and sizes of the two regions of interest are the same in image 1 and image 2 respectively, the positions and sizes of the regions of interest in the road images of the two frames can be considered to correspond.

[0060] Regions of interest (the two regions of interest can be denoted as ROI1 and ROI2) (301 and 302) are extracted in image 1 and image 2, respectively, and ROI1 and ROI2 are converted into grayscale images, respectively, to obtain grayscale image 1 and grayscale image 2. Optionally, a window function (e.g., a Hanning window) (203) can be applied to grayscale image 1 (303) and grayscale image 2 (304), respectively, to reduce edge effects. The processed results of grayscale image 1 and grayscale image 2 are subjected to Fourier transform processing (204), such as 2D FFT or Discrete Fourier Transform (DFT), respectively, and in this example, FFT (2D FFT) is used for explanation. The second result (e.g., the processed result corresponding to grayscale image 2) is selected and complex conjugated (205), and a normalized cross process (206) is performed on each corresponding pixel point in processed grayscale image 1 and grayscale image 2, which may involve performing element-by-element normalization on the two images after Fourier transformation and calculating a cross power spectrum. In some embodiments, the processed images are further subjected to an inverse Fourier transform process to obtain a cross-correlation diagram. Here, the inverse Fourier transform process may be, for example, a 2D inverse fast Fourier transform (IFFT) (207) or an inverse discrete Fourier transform (IDFT), and in this example, an IFFT (2D IFFT) is used as an example.Optionally, an FFT shift (208) and a picture order count (POC) reflection (POC Image) (209) are performed on the cross-correlation diagram, and the value of each pixel point of the processed image is searched to find the maximum value (i.e., peak value, for example, the white circle point in the last drawing of FIG. 3 ) (305) (this process is called Peak Position Search (210)), and the peak position is determined, and the pixel coordinate of the peak position represents the amount of movement between the two images (also called the offset amount of the corresponding vanishing points in the two images), i.e., the phase offset information (or phase offset amount) between the road images of the two frames.

[0061] Optionally, a sub-center estimation (this process is called "sub-pixel estimation (or secondary pixel estimation)" or "subcenter estimation") is performed by obtaining the centroid (or mass centroid) from pixels near the peak position to determine the Peak Position Centroid (211) to determine the Sub-pixel Correlation Position (212).

[0062] Optionally, the size of the region of interest may be a size convenient for Fourier transform, for example, the number of pixels on each side of the region of interest is 2 n . For example, the size of the region of interest may be 1024*1024. Optionally, a size reduction process may be performed on the region of interest, for example, reducing it to a size of 256*256, thereby reducing the amount of data processing. When a size reduction process is performed on the region of interest, after obtaining the peak position coordinates, i.e., after obtaining the phase offset information between the road images of the two frames, the phase offset information is further processed based on a reduction ratio to obtain phase offset information of the same size as image 1 and image 2.

[0063] In some embodiments of the present invention, after determining phase offset information between road images of two frames, this offset is caused by a change in attitude between the traveling device and the ground due to acceleration or deceleration of the traveling device or unevenness of the road surface. Such a change in attitude represents a change in the relative attitude between the traveling device and the ground, and may be a change in pitch attitude. The phase offset information represents an offset amount between corresponding vanishing points in the two images, and part of this offset amount is caused by the traveling device's horizontal movement, and the other part is caused by a change in pitch attitude of the traveling device. In some embodiments, the electronic device needs to determine attitude change information of the traveling device based on the phase offset information.

[0064] In some embodiments, the electronic device converts the phase offset information according to a conversion relationship to obtain attitude change information of the traveling device, which may refer to attitude change information in the vertical direction of the traveling device (also referred to as pitch attitude change information).

[0065] In an embodiment of the present invention, the electronic device can predetermine reference attitude information, and the reference attitude information is determined based on at least two frames of historical road images. For example, the at least two frames of historical road images may be road images collected when the traveling device is traveling on a flat road at a constant or approximately constant speed. The attitude information of the traveling device is determined based on the at least two frames of historical road images, and the attitude information is used as reference attitude information (also referred to as reference attitude information of the traveling device). Subsequently, all attitude change information obtained by the above technical solution of the present invention is based on the reference attitude information, and the attitude change information (which may be a vector) is added to the reference attitude information to obtain the attitude information of the traveling device.

[0066] In some alternative embodiments of the present invention, the method further comprises updating calibration information of the image acquisition component based on the attitude change information or attitude information of the traveling equipment.

[0067] In an embodiment of the present invention, the electronic device can update calibration information of the image collection component based on the posture change information or the posture information of the traveling device, and the calibration information can be, for example, a homography matrix. The updated calibration information is used for bird's-eye view transformation, which includes bird's-eye view transformation from at least one of image information such as lanes, target objects, etc. to bird's-eye view information.

[0068] Based on the above embodiments, an embodiment of the present invention further provides an image processing device. Figure 4 is an exemplary structural diagram 1 of the configuration of an image processing device of an embodiment of the present invention. As shown in Figure 4, the device comprises an acquisition unit 21, an attitude offset sensing unit 22 and an attitude determination unit 23, where: The acquisition unit 21 is configured to acquire at least two frames of road images through an image collection component provided in the traveling equipment, the attitude offset sensing unit 22 is configured to determine attitude change information of the traveling equipment between two frames of road images of the at least two frames of road images using a phase-only correlation method, and the attitude determination unit 23 is configured to determine attitude information of the traveling equipment based on the attitude change information and reference attitude information of the traveling equipment.

[0069] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames, and to determine attitude change information of the traveling equipment between the road images of the two frames based on the phase offset information.

[0070] In some alternative embodiments of the present invention, the acquisition unit 21 is further configured to determine one or at least two regions of interest in the road image of each frame, and at least one of the one or at least two regions of interest has at least one of the following characteristics: overlapping with the horizon line; and including a vanishing point.

[0071] In some alternative embodiments of the invention, there are no moving objects in the region of interest, or the proportion of the moving objects occupying the region of interest is less than or equal to a first threshold.

[0072] In some alternative embodiments of the present invention, when the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest, wherein the first region of interest includes the vanishing point.

[0073] In some alternative embodiments of the present invention, the size of the second region of interest is smaller than the size of the first region of interest.

[0074] In some alternative embodiments of the present invention, the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps with the first region of interest.

[0075] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on an area of ​​interest within the road image of each frame of the road images of the two frames.

[0076] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames when the first region of interest includes a moving object.

[0077] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to, when a moving object is included in the area of ​​interest in the road image of each of the two frames, determine first phase offset information between the road images of the two frames based on the area of ​​interest in the road image of the two frames, detect the proportion of the area where the moving object is located in each of the areas of interest in the road image of the two frames, and determine phase offset information between the road images of the two frames based on the proportion and the first phase offset information.

[0078] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine second phase offset information between the road images of the two frames based on the corresponding groups of regions of interest in the road images of the two frames when the road images of each frame include at least two regions of interest, where the positions of each region of interest in each group in the respective road images correspond, and select one second phase offset information from the at least two second phase offset information as the phase offset information between the road images of the two frames, or determine a median phase change information among the at least two second phase offset information and determine the median phase change information as the phase offset information between the road images of the two frames, or determine an average value of the at least two second phase offset information and determine the average value as the phase offset information between the road images of the two frames.

[0079] In some alternative embodiments of the present invention, the at least one region of interest is symmetrical with respect to the horizon.

[0080] In some alternative embodiments of the present invention, the pixel points on each side of the at least one region of interest are 2 n , where n is a positive integer.

[0081] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to: extract sub-images corresponding to regions of interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image; perform grayscale processing on the first sub-image and the second sub-image, respectively; obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image; perform Fourier transform processing on the first grayscale image and the second grayscale image, respectively; perform normalized cross processing on corresponding pixel points in the processed first grayscale image and the processed second grayscale image; obtain a processed image; perform inverse Fourier transform processing on the processed image; determine a peak position based on the value of each pixel point in the processed image; and determine phase offset information between the road images of the two frames based on the peak position.

[0082] In some alternative embodiments of the present invention, as shown in FIG. 5, the device further comprises an update unit 24 configured to update calibration information of the image collection component based on the posture change information or posture information of the traveling equipment.

[0083] In an embodiment of the present invention, the apparatus is applied to an electronic device, and the acquisition unit 21, the attitude offset sensing unit 22, the attitude determination unit 23 and the update unit 24 in the apparatus can all be realized by a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA) in practical applications.

[0084] Although the image processing device according to the above embodiment only describes the division of each of the above program modules as an example of image processing, in actual applications, the above processes may be assigned to different program modules as needed, i.e., the internal structure of the device may be divided into different program modules to perform all or part of the above processes. Furthermore, the image processing device according to the above embodiment belongs to the same concept as the image processing method embodiment, and its implementation process may refer to the method embodiment, and will not be described again here.

[0085] An embodiment of the present invention further provides an electronic device, and FIG. 6 is an exemplary structural diagram of the hardware configuration of the electronic device of the embodiment of the present invention. As shown in FIG. 6, the electronic device comprises a memory 32, a processor 31, and a computer program stored in the memory 32 and executable by the processor 31, and the processor 31 executes the program to realize the steps of the image processing method described in the embodiment of the present invention.

[0086] Optionally, the electronic device may further include a user interface 33 and a network interface 34. Here, the user interface 33 may include a display, a keyboard, a mouse, a trackball, a click wheel, keystrokes, buttons, a touchpad or a touch screen, etc.

[0087] Optionally, each component in the electronic device is coupled via a bus system 35. It can be understood that the bus system 35 is used to realize connection communication between these components. In addition to a data bus, the bus system 35 further includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, various buses are labeled as the bus system 35 in FIG. 6 .

[0088] It will be understood that memory 32 may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Here, nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM), and magnetic surface memory may be magnetic disk memory or magnetic tape memory. Volatile memory may be random access memory (RAM) used as an external cache.By way of illustrative, but not limiting example, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), SyncLink dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM), etc. Memory 32 as described in embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0089] The methods disclosed in the above embodiments of the present invention can be applied to or realized by the processor 31. The processor 31 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be performed by an integrated logic circuit of hardware in the processor 31 or by instructions in software form. The processor 31 can be a general-purpose processor, a DSP, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The processor 31 can implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly executed by a hardware decoding processor or by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in the memory 32, and the processor 31 reads information in the memory 32 and performs the steps of the above method in cooperation with the hardware.

[0090] In an exemplary embodiment, the electronic device may be implemented with one or at least two Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic elements to perform the above methods.

[0091] In an exemplary embodiment, an embodiment of the present invention further provides a computer-readable storage medium, such as a memory 32, containing a computer program, which can be executed by a processor 31 of the electronic device to perform the steps of the above-described method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM, or may be various devices including any of the above memories or any combination thereof.

[0092] A computer-readable storage medium according to an embodiment of the present invention stores a computer program for causing a processor to execute steps of an image processing method according to an embodiment of the present invention.

[0093] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device, and may be a volatile or non-volatile storage medium. A computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable CD-ROMs (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encoding devices (e.g., punch cards or slotted protruding structures on which instructions are stored), and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a momentary signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted by a wire.

[0094] An embodiment of the present invention further provides a computer program, which includes computer-readable code that, when read and executed by a computer, implements some or all of the steps of the method in any one of the embodiments of the present invention.

[0095] The methods disclosed in the several method embodiments provided in the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0096] The features disclosed in the several product embodiments provided by the present invention may be combined in any non-conflicting manner to obtain new product embodiments.

[0097] The features disclosed in the several method or apparatus embodiments provided in the present invention may be combined in any non-conflicting manner to obtain new method or apparatus embodiments.

[0098] In some embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be realized in other ways. The device embodiments described above are merely exemplary. For example, the division of the units is merely a division of logical functions. In actual implementation, other division methods may be used. For example, at least two units or components may be combined or integrated into another system, and some features may be ignored or not implemented. In addition, the couplings or direct couplings or communication connections between the components shown or discussed may be indirect couplings or communication connections via several interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0099] The units described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed across at least two network units, and some or all of the units therein may be selected according to actual needs to achieve the objectives of the technical solutions in the embodiments of the present invention.

[0100] Furthermore, the functional units in each embodiment of the present invention may all be integrated into a single processing unit, or each unit may be separated into an independent unit, or two or more units may be integrated into a single unit, and the above-mentioned integrated units may be realized as hardware or as a combination of hardware and software.

[0101] As can be understood by those skilled in the art, all or part of the steps of the above method embodiments can be performed by hardware associated with program instructions, and the program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are performed, and the storage medium includes a medium that can store program code, such as a mobile storage, a ROM, a RAM, a magnetic memory, or an optical disk.

[0102] Alternatively, when the above-described integrated units in the embodiments of the present invention are realized in the form of software functional modules and sold or used as independent products, they can be stored in a single computer-readable storage medium. Based on this understanding, the essential parts of the technical solutions of the present invention, i.e., the parts that contribute to the prior art, can be embodied in the form of a software product, and the computer software product is stored in a single storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage, ROM, RAM, magnetic disk, or optical disk.

[0103] The above content is merely an embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any modifications or replacements that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be governed by the protection scope of the claims.

Claims

1. An image processing method, comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining, using a phase-only correlation method, attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images; determining attitude information of the traveling equipment by adding the attitude change information and reference attitude information determined based on historical road information of at least two frames collected while the traveling equipment was traveling on a flat road surface.

2. Determining posture change information of the traveling equipment between two frames of road images among the at least two frames of road images using the phase-only correlation method includes: determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames; and determining attitude change information of the traveling equipment between the road images of the two frames based on the phase offset information. The image processing method according to claim 1 .

3. The region of interest in the road image for each frame is by determining one or at least two regions of interest in the road image of each frame, wherein at least one of the one or at least two regions of interest has at least one of the following characteristics: overlapping with a horizon line; and including a vanishing point. The image processing method according to claim 2 .

4. There are no moving objects in the region of interest, or a proportion of the moving objects occupying the region of interest is less than or equal to a first threshold; The image processing method according to claim 2 .

5. when the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest; the first region of interest includes the vanishing point; The image processing method according to claim 3 .

6. The size of the second region of interest is smaller than the size of the first region of interest; the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps the first region of interest; The image processing method according to claim 5 .

7. Determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames includes: determining phase offset information between the road images of the two frames based on a region of interest in the road image of each frame of the road images of the two frames; The image processing method according to claim 4.

8. The first region of interest includes a moving object, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes: determining phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames; The image processing method according to claim 5 .

9. An image processing method, comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining, using a phase-only correlation method, attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images; determining attitude information of the traveling device based on the attitude change information and reference attitude information of the traveling device; Determining posture change information of the traveling equipment between two frames of road images among the at least two frames of road images using the phase-only correlation method includes: determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames; determining, based on the phase offset information, attitude change information of the traveling equipment between the road images of the two frames; The road images of the two frames each include a moving object in an area of ​​interest in the road image of each frame, and determining phase offset information between the road images of the two frames based on the area of ​​interest in the road images of the two frames includes: determining first phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames; detecting a proportion of an area where the moving object is located in a region of interest in road images of each of the two frames, and determining phase offset information between the road images of the two frames based on the proportion and the first phase offset information; Image processing methods.

10. An image processing method, comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images using a phase-only correlation method; determining attitude information of the traveling device based on the attitude change information and reference attitude information of the traveling device; Determining posture change information of the traveling equipment between two frames of road images among the at least two frames of road images using the phase-only correlation method includes: determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames; determining, based on the phase offset information, attitude change information of the traveling equipment between the road images of the two frames; The road image of each frame includes at least two regions of interest, and determining phase offset information between the road images of the two frames based on the regions of interest in the road images of the two frames includes: determining second phase offset information between the road images of the two frames based on the regions of interest of each corresponding group in the road images of the two frames, where the positions of the regions of interest of each group in the respective road images correspond to each other; selecting one piece of second phase offset information from at least two pieces of second phase offset information as the phase offset information between the road images of the two frames, or determining median phase change information in the at least two pieces of second phase offset information and determining the median phase change information as the phase offset information between the road images of the two frames, or determining an average value of the at least two pieces of second phase offset information and determining the average value as the phase offset information between the road images of the two frames, Image processing methods.

11. The region of interest is symmetrical with respect to the horizon. The image processing method according to any one of claims 2 to 10.

12. The pixel points on each side of the region of interest are 2 n , where n is a positive integer. The image processing method according to any one of claims 2 to 10.

13. Determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames includes: extracting sub-images corresponding to regions of interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image; performing grayscale processing on the first sub-image and the second sub-image, respectively, to obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image; performing a Fourier transform process on the first grayscale image and the second grayscale image, respectively, and performing a normalized cross process on each corresponding pixel point in the processed first grayscale image and the processed second grayscale image to obtain a processed image; performing an inverse Fourier transform process on the processed image, determining a peak position based on the value of each pixel point of the processed image, and determining phase offset information between the road images of the two frames based on the peak positions; The image processing method according to claim 2 .

14. The image processing method includes: and further comprising updating calibration information of the image acquisition component based on the attitude change information or the attitude information of the traveling device. The image processing method according to any one of claims 1 to 10.

15. An image processing apparatus comprising: an acquisition unit; an attitude offset sensing unit; and an attitude determination unit; The acquisition unit is configured to acquire at least two frames of road images via an image acquisition component provided on a traveling device; The attitude offset sensing unit is configured to determine attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images using a phase-only correlation method; The posture determination unit is configured to determine posture information of the traveling equipment by adding the posture change information and reference posture information determined based on historical road information of at least two frames collected when the traveling equipment is traveling on a flat road surface.

16. A computer-readable storage medium storing a computer program for causing a processor to execute the image processing method according to any one of claims 1 to 10.

17. 11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the processor executing the computer program to realize the image processing method according to any one of claims 1 to 10.

18. A computer program comprising computer readable code for causing a processor of a device to carry out the image processing method of any one of claims 1 to 10.

Images