Image processing method and device, electronic device, storage medium, computer program, and computer program product

Abstract

The embodiments of the present invention disclose an image processing method and its device, electronic device, storage medium, computer program, and computer program product, which includes: acquiring at least two frames of road images through an image acquisition component provided in a traveling device, determining posture change information of the traveling device between two frames of the road images of the at least two frames using a phase-only correlation method, and determining posture information of the traveling device based on the posture change information and reference posture information of the traveling device.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is based on a Chinese patent application filed on March 24, 2022, bearing application number 202210303559.6 and entitled "Image processing method and apparatus, electronic device, and storage medium", and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer vision technology, and in particular to an image processing method and apparatus, an electronic device, a storage medium, a computer program, and a computer program product. [Background technology]

[0003] An accurate and robust calibration system is very important for an autonomous driving system, as it determines the reference coordinate system of the measurements output by each sensor and improves the accuracy and consistency during the vehicle's movement. However, the attitude of the vehicle relative to the ground changes due to the unevenness of the road surface and the acceleration and deceleration of the vehicle.

[0004] This problem is usually solved in two technical aspects: software and hardware. In the hardware aspect, a high-precision inertial navigation sensor is usually used to provide accurate position and orientation information, but this method is usually relatively expensive. In the software aspect, a feature point matching method is usually used to calculate the position and orientation change, but this method is relatively time-consuming. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the existing technical problems, embodiments of the present invention provide an image processing method and apparatus, an electronic device, a storage medium, a computer program, and a computer program product. [Means for solving the problem]

[0006] To achieve the above objectives, the technical solutions of the embodiments of the present invention are realized as follows:

[0007] An embodiment of the present invention provides an image processing method, the method comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining attitude change information of the traveling equipment between two frames of road images of the at least two frames of road images using a phase-only correlation method; Determining attitude information of the traveling equipment based on the attitude change information and reference attitude information of the traveling equipment.

[0008] An embodiment of the present invention further provides an image processing apparatus, the apparatus comprising: an acquisition unit, an attitude offset sensing unit, and an attitude determination unit, wherein: The acquisition unit is configured to acquire at least two frames of road images via an image acquisition component provided on a traveling device; The attitude offset sensing unit is configured to determine attitude change information of the traveling equipment between two frames of road images of the at least two frames of road images by using a phase-only correlation method; The attitude determination unit is configured to determine attitude information of the traveling equipment based on the attitude change information and reference attitude information of the traveling equipment.

[0009] An embodiment of the present invention further provides a computer readable storage medium having stored thereon a computer program for causing a processor to perform the steps of the method according to an embodiment of the present invention.

[0010] An embodiment of the present invention further provides an electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the processor executing the program to implement steps of the method according to the embodiment of the present invention.

[0011] An embodiment of the present invention provides a computer program comprising computer readable code, which, when read and executed by a computer, causes the computer to perform some or all of the steps of the method in any one of the embodiments of the present application.

[0012] An embodiment of the present invention provides a computer program product, the computer program product including a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being configured to, when read and executed by a computer, cause the computer to perform some or all of the steps of a method in any one of the embodiments of the present application. Effect of the Invention

[0013] The embodiments of the present invention provide an image processing method and its device, electronic device, storage medium, computer program, and computer program product, the method including: acquiring at least two frames of road images through an image acquisition component provided in a traveling device; determining attitude change information of the traveling device between two frames of road images among the at least two frames of road images using a phase-only correlation method; and determining attitude information of the traveling device based on the attitude change information and the reference attitude information of the traveling device. According to the technical solution of the embodiments of the present invention, the attitude change information between road images is determined using a phase-only correlation method. Compared with a method employing a high-precision inertial navigation positioning sensor, the embodiments of the present invention can reduce costs while obtaining accurate position and attitude information. Compared with a method employing a feature point matching method to calculate position and attitude information, the embodiments of the present invention can reduce the amount of calculation and can be applied to a monocular visual sensor. [Brief description of the drawings]

[0014] [Figure 1] 1 is an exemplary flowchart 1 of an image processing method according to an embodiment of the present invention. [Diagram 2] 4 is a schematic diagram of a process for processing phase offset information in the image processing method according to an embodiment of the present invention. FIG. [Diagram 3] 4 is a schematic diagram of phase offset information in the image processing method according to the embodiment of the present invention; FIG. [Figure 4] 1 is an exemplary structural diagram of an image processing device according to an embodiment of the present invention; [Diagram 5] FIG. 2 is an exemplary structural diagram of an image processing device according to an embodiment of the present invention. [Figure 6] FIG. 2 is an exemplary structural diagram of a hardware configuration of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings necessary in the embodiments are briefly introduced above. The drawings herein are incorporated in and constitute a part of this specification, and these drawings are for illustrating the embodiments according to the present application, and are used together with this specification to explain the technical solutions of the embodiments of the present invention. The drawings only show a part of the embodiments of the present invention, and therefore should not be regarded as limiting the scope of the present application, and it should be understood that those skilled in the art can also obtain other related drawings according to these drawings without creative efforts.

[0016] In the following, the present application will be explained in more detail with reference to the drawings and specific embodiments.

[0017] In the embodiments of the present invention, the terms "comprise", "include" or any other variations thereof are intended to cover a non-exclusive inclusiveness, whereby a method or apparatus including a set of elements not only includes the elements explicitly described, but also includes other elements not explicitly listed, or includes elements inherent to implementing the method or apparatus. Unless otherwise limited, an element defined by the phrase "comprises" does not exclude the presence of another related element (e.g., a step in a method or a unit in an apparatus, where an example unit may be a partial circuit, a partial processor, a partial program or software, etc.) in the method or apparatus including the element.

[0018] For example, an image processing method according to an embodiment of the present invention includes a series of steps, but the image processing method according to an embodiment of the present invention is not limited to the steps described. Similarly, an image processing device according to an embodiment of the present invention includes a series of modules, but the device according to an embodiment of the present invention is not limited to including the modules explicitly described, and may further include a module for obtaining related information or a module for processing based on the information.

[0019] The term "and / or" in this specification describes the relationship of the related objects and indicates that three relationships may exist, for example, A and / or B can represent three cases: A exists independently, A and B both exist, and B exists independently. In addition, the term "at least one" in this specification indicates any one of a plurality, or any combination of at least two of a plurality, for example, including at least one of A, B, and C indicates including any one or more elements selected from the set consisting of A, B, and C.

[0020] An embodiment of the present invention provides an image processing method. Figure 1 is an exemplary flow chart 1 of the image processing method of an embodiment of the present invention, as shown in Figure 1, the method includes the following steps:

[0021] In step S101, at least two frames of road images are acquired via an image acquisition component provided in the traveling equipment.

[0022] In step S102, a phase-only correlation method is used to determine information on a change in attitude of the traveling equipment between two frames of road images among the at least two frames of road images.

[0023] In step S103, attitude information of the traveling equipment is determined based on the attitude change information and the reference attitude information of the traveling equipment.

[0024] In some embodiments of the present invention, the traveling device may be an autonomous vehicle, a vehicle equipped with an advanced driver assistance system, a robot, etc. The image processing method of the embodiments of the present invention is applied to an electronic device, and the electronic device may be an in-vehicle device, a cloud platform, or other computer device. In some embodiments, the in-vehicle device may be a thin client, a thick client, a microprocessor-based system, a small computer system, etc. implemented in a vehicle, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system, etc. In some possible embodiments, the image processing method may be realized by a manner in which a processor calls computer-readable instructions stored in a memory.

[0025] In an embodiment of the present invention, the on-board device can be communicatively connected to a sensor of the vehicle, a positioning device, etc., and the on-board device can obtain data collected by the sensor of the vehicle and geographical location information reported by the positioning device, etc., via the communication connection. In some embodiments, the sensor of the vehicle may be at least one of a millimeter wave radar, a laser radar (LIDAR), a camera, etc., and the positioning device may be a device that provides a positioning service based on at least one positioning system of a Global Positioning System (GPS), a Beidou satellite navigation system, or a Galileo satellite navigation system.

[0026] In one example, the in-vehicle device may be an advanced driving assistance system (ADAS), where the ADAS is installed in a vehicle and can acquire real-time position information of the vehicle from a positioning device of the vehicle, and the ADAS can also acquire image data, radar data, etc. representing vehicle surrounding environment information from a sensor of the vehicle. Here, optionally, the ADAS can transmit vehicle driving data including real-time position information of the vehicle to a cloud platform, and in this way, the cloud platform can receive at least one of the real-time position information of the vehicle, the image data representing vehicle surrounding environment information, and the radar data.

[0027] In an embodiment of the present invention, road images are acquired via an image collection component (i.e., the above-mentioned sensor such as a camera) provided on the traveling equipment, and the image collection component collects road images or environmental images around the vehicle in real time as the traveling equipment moves.

[0028] According to the technical solution of an embodiment of the present invention, a phase-only correlation method is used to determine pose change information between road images; compared with the method of adopting a high-precision inertial navigation positioning sensor, the embodiment of the present invention can reduce costs while obtaining accurate position and attitude information; compared with the method of calculating position and attitude information by adopting a feature point matching method, the embodiment of the present invention can reduce the amount of calculation and is applicable to monocular visual sensors.

[0029] In some alternative embodiments of the present invention, determining posture change information of the moving equipment between two road images of the at least two frames using the phase-only correlation method includes determining phase offset information between the road images of the two frames based on a region of interest in the road images of the two frames, and determining posture change information of the moving equipment between the road images of the two frames based on the phase offset information.

[0030] In the field of image processing, a Region Of Interest (ROI) is a region selected from an image (i.e., a road image in the embodiment of the present invention), which is a key point of interest in image analysis or processing. In some embodiments, the region of interest can be determined in a square, circle, ellipse, irregular polygon, etc. In the embodiment of the present invention, a square is taken as an example to determine the region of interest in the road image.

[0031] In an embodiment of the present invention, by setting a region of interest (ROI) in a road image, phase offset information between road images is determined by a phase-only correlation method based on the ROI in each road image, and then, based on the phase offset information, posture change information of the traveling equipment between the road images of two frames is determined. Compared to a method of calculating position and posture information by adopting a feature point matching method, the embodiment of the present invention can significantly reduce the amount of calculations.

[0032] In some alternative embodiments, the region of interest in the road image of each frame is determined by the following method: determining one or at least two regions of interest in the road image of each frame, at least one of the one or at least two regions of interest having at least one of the following characteristics: overlapping with a horizon line and including a vanishing point.

[0033] In an embodiment of the present invention, one or at least two regions of interest are set in a road image of each frame, where at least one region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point. In some embodiments, when a road image includes one region of interest, the region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point. When a road image includes at least two regions of interest, each region of interest has at least one of the following characteristics: overlapping with the horizon line and including a vanishing point.

[0034] Optionally, each region of interest of the at least two regions of interest may overlap with at least one other region of interest.

[0035] Here, the size and aspect ratio of the set region of interest may be set according to actual needs, and the embodiment of the present invention is not limited thereto.

[0036] In a normal situation, the road image is acquired through an image acquisition component provided on a traveling device, and the traveling device generally travels on a road, that is, the road image generally includes lanes on the road, and the vanishing point represents a parallel line on the actual road, for example, a point where the lanes or road edge lines visually intersect in the road image, and the region of interest in the embodiment of the present invention includes the vanishing point.

[0037] In some alternative embodiments, the at least one region of interest is symmetrical with respect to the horizon.

[0038] In some alternative embodiments, the pixel points on each side of the at least one region of interest are a power of 2, where n is a positive integer.

[0039] To facilitate image processing, in an embodiment of the present invention, the size of the at least one region of interest is a size that can be easily processed by image processing, for example, when the image processing process includes a Fast Fourier Transform (FFT), the pixel points on each side of the at least one region of interest are 2 to the power of 2, where n is a positive integer.

[0040] In some alternative embodiments, there are no moving objects in the region of interest, or the percentage of the moving objects occupying the region of interest is less than or equal to a first threshold.

[0041] In an embodiment of the present invention, the selected region of interest does not include a moving object, or the ratio of the moving object included in the selected region of interest to the region of interest is small, for example, the ratio of the moving object occupying the region of interest is equal to or less than a first threshold, so that the calculated phase offset information between the road images of two frames is not influenced by the moving object, or is influenced by the moving object only slightly, and the obtained phase offset information and the attitude offset information of the traveling equipment are more accurate. Here, the value of the first threshold can be set according to the actual situation, and the embodiment of the present invention is not limited thereto. In some embodiments, the moving object may be another traveling equipment (e.g., another vehicle) in the road image.

[0042] In some alternative embodiments, when the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest, where the first region of interest includes the vanishing point.

[0043] In another embodiment, the first region of interest is also called a main region of interest (Main ROI), and the second region of interest is also called an auxiliary region of interest (Auxiliary ROI). In a different scenario, the electronic device can determine phase offset information between the road images of the two frames based on at least one of the first region of interest, the second region of interest in the road images of the two frames.

[0044] Optionally, a size of the second region of interest is smaller than a size of the first region of interest.

[0045] Optionally, the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps with the first region of interest.

[0046] In some alternative embodiments of the present invention, determining phase offset information between the road images of the two frames based on an area of ​​interest in the road images of the two frames includes determining phase offset information between the road images of the two frames based on an area of ​​interest in the road image of each of the road images of the two frames.

[0047] In an embodiment of the present invention, when there is no moving object in the region of interest, or when the proportion of moving objects occupying the region of interest is below a first threshold, that is, when the phase offset information between the road images of two frames is not affected by moving objects or the influence of moving objects is small, the phase offset information between the road images of the two frames can be calculated based on the region of interest in the road image of each frame.

[0048] In some alternative embodiments of the present invention, a moving object is present in the first region of interest, and determining phase offset information between the road images of the two frames based on a region of interest in the road images of the two frames includes determining phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames.

[0049] In an embodiment of the present invention, if a moving object exists in the first region of interest (e.g., main region of interest), since the setting positions of the second region of interest and the first region of interest are different, it is highly likely that the moving object does not exist in the second region of interest, and the phase offset information between the road images of the two frames is determined based on the second region of interest (e.g., auxiliary region of interest) in the road images of the two frames. In another embodiment, if a moving object does not exist in the first region of interest (e.g., main region of interest), the phase offset information between the road images of the two frames can be determined based only on the first region of interest (e.g., main region of interest), or the phase offset information between the road images of the two frames can be determined based on the first region of interest and the second region of interest.

[0050] In some alternative embodiments, if the region of interest in the first road image contains a moving object, or if the proportion of moving objects occupying the region of interest in the first road image is equal to or greater than a second threshold, the first road image is not used to determine phase offset information between the first road image and other road images, i.e. the first road image is not used to determine phase offset information between the first road image and other road images, i.e. the first road image is not used to calculate the phase offset information, thereby excluding the influence of moving objects in the region of interest of the road image on the phase offset information obtained by the calculation.

[0051] In some alternative embodiments, a moving object is present in a region of interest in the road image of each of the two frames, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes determining first phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames, detecting a proportion of the region where the moving object is located in the region of interest in each of the road images of the two frames, and determining phase offset information between the road images of the two frames based on the proportion and the first phase offset information.

[0052] In an embodiment of the present invention, when a moving object exists in both of the regions of interest in the road images of two frames for determining the phase offset information, the calculated phase offset information between the road images of the two frames (herein referred to as first phase offset information) is calibrated to calibrate the effect of the moving object on the calculated phase offset information. For example, after calculating the first phase offset information based on the regions of interest in the road images of the two frames, the electronic device can detect the ratio of the moving object in the region of interest, for example, detect the detection frame of the moving object to determine the overlap region between the moving object and the region of interest (if the moving object is large, the whole of the moving object may not be within the region of interest), and further determine the ratio of the size (or area) of the overlap region to the region of interest, and determine the phase offset information between the road images of the two frames based on the ratio and the first phase offset information.

[0053] Optionally, the ratio may be multiplied with the first phase offset information to obtain phase offset information between the road images of the two frames.

[0054] In some alternative embodiments, the road image of each frame includes at least two regions of interest, and determining phase offset information between the road images of the two frames based on the regions of interest in the road images of the two frames includes determining second phase offset information between the road images of the two frames based on each corresponding group of regions of interest in the road images of the two frames, wherein the positions of each region of interest in each group in the respective road images correspond, and selecting one second phase offset information from the at least two second phase offset information as the phase offset information between the road images of the two frames, or determining a median phase change information in the at least two second phase offset information and determining the median phase change information as the phase offset information between the road images of the two frames, or determining an average value of the at least two second phase offset information and determining the average value as the phase offset information between the road images of the two frames.

[0055] In an embodiment of the present invention, when a road image includes at least two regions of interest, for example, road image 1 includes three regions of interest, and road image 2 includes three corresponding regions of interest. Then, each region of interest in road image 1 can form one group of regions of interest with a corresponding region of interest in road image 2, and the positions of each region of interest in each group in each road image correspond to each other. Then, for each group of regions of interest, second phase offset information between road images of two frames can be calculated. As one embodiment, one piece of second phase offset information is selected from at least two pieces of second phase offset information, and can be used as the phase offset information between road images of two frames in the calculation of the posture information of the subsequent traveling equipment. As a second embodiment, a median phase offset information in at least two pieces of second phase offset information can be determined, and the median phase change information can be determined as the phase offset information between road images of the two frames, and in the case of the median phase offset information, the second phase offset information located in the middle after rearranging the at least two pieces of second phase offset information in descending or ascending order is the median phase offset information. As a third embodiment, at least two pieces of second phase offset information are added together to find an average value, and the average value obtained by calculation can be used as the phase offset information between the road images of two frames.

[0056] In the above third embodiment, before the process of adding at least two pieces of second phase offset information to obtain an average value, the method further includes removing abnormal second phase offset information from the at least two pieces of second phase offset information, and adding the second phase offset information after the abnormal values ​​are removed to calculate an average value to obtain phase offset information between road images of two frames. Here, in general, the second phase offset information calculated based on each region of interest should be similar, that is, the difference between each piece of second phase offset information does not exceed a certain threshold. Then, the abnormal value may be second phase offset information whose difference with any second phase offset information exceeds the threshold, that is, second phase offset information with a large difference is considered as the abnormal value and needs to be removed.

[0057] In some alternative embodiments of the present invention, determining phase offset information between the road images of the two frames based on an area of ​​interest in the road images of the two frames includes: extracting sub-images corresponding to areas of interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image; performing grayscale processing on the first sub-image and the second sub-image, respectively, to obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image; performing Fourier transform processing on the first grayscale image and the second grayscale image, respectively, and performing normalized cross processing on corresponding pixel points in the processed first grayscale image and the processed second grayscale image to obtain a processed image; performing an inverse Fourier transform processing on the processed image, determining a peak position based on a value of each pixel point of the processed image, and determining phase offset information between the road images of the two frames based on the peak position.

[0058] Here, the normalized cross processing refers to a process of performing a normalization process on each pixel of the first grayscale image and the second grayscale image that have been Fourier transformed, and calculating the cross power spectrum for each pixel of the two normalized images, and then performing an inverse Fourier transform process on the processed images, which is also called a normalized cross-correlation process.

[0059] In some embodiments, referring to Figures 2 and 3, taking road images of two frames as an example, assuming that the road images of each frame each have one region of interest, for example, a rectangular framed area in image 1 is the region of interest in image 1 (201), and a rectangular framed area in image 2 (202) is the region of interest in image 2, and since the positions and sizes of the two regions of interest are the same in images 1 and 2, respectively, the positions and sizes of the regions of interest in the road images of the two frames can be considered to correspond.

[0060] Regions of interest (the two regions of interest can be denoted as ROI1 and ROI2) (301 and 302) in image 1 and image 2 are extracted, respectively, and ROI1 and ROI2 are converted into grayscale images, respectively, to obtain grayscale image 1 and grayscale image 2. Optionally, a window function (e.g., Hanning Window) (203) can be applied to grayscale image 1 (303) and grayscale image 2 (304), respectively, to reduce edge effects. Fourier transform processing (204), such as 2D FFT or Discrete Fourier Transform (DFT), is performed on the processed results of grayscale image 1 and grayscale image 2, respectively, and FFT (2D FFT) is taken as an example in this example. The second result (e.g., the processed result corresponding to grayscale image 2) is selected to perform Complex Conjugate (205), and a Normalized Cross process (206) is performed on each corresponding pixel point in the processed grayscale image 1 and grayscale image 2, which may be element-wise normalization of the two images after Fourier transformation to calculate a cross power spectrum. In some embodiments, the processed images are further subjected to an inverse Fourier transform process to obtain a cross-correlation diagram. Here, the inverse Fourier transform process may be, for example, a 2D inverse fast Fourier transform (IFFT) (207) or an inverse discrete Fourier transform (IDFT), and in this example, IFFT (2D IFFT) is taken as an example.Optionally, an FFT shift (208) and a Picture Order Count (POC) reflection (POC Image) (209) are performed on the cross-correlation diagram, and the value of each pixel point of the processed image is searched to find the maximum value (i.e., peak value, e.g., the white circle point in the last drawing of FIG. 3 ) (305) (this process is called Peak Position Search (210)), and a peak position is determined, and the pixel coordinate of the peak position represents the amount of movement between the two images (also called the offset amount of the corresponding vanishing points in the two images), i.e., the phase offset information (or phase offset amount) between the road images of the two frames.

[0061] Optionally, a sub-center estimation (this process is called “sub-pixel estimation (or secondary pixel estimation)” or “subcenter estimation”) is performed by taking the centroid (or mass centroid) from pixels near the peak position to determine the Peak Position Centroid (211) to determine the Sub-pixel Correlation Position (212).

[0062] Optionally, the size of the region of interest may be a size convenient for Fourier transformation, for example, the number of pixels on each side of the region of interest is 2 n . For example, the size of the region of interest may be 1024*1024. Optionally, a size reduction process may be performed on the region of interest, for example, reducing the size to 256*256, thereby reducing the amount of data processing. When a size reduction process is performed on the region of interest, after obtaining the peak position coordinate, i.e., after obtaining the phase offset information between the road images of the two frames, the phase offset information is further processed based on the reduction ratio to obtain phase offset information of the same size as image 1 and image 2.

[0063] In the embodiment of the present invention, after determining the phase offset information between the road images of the two frames, this offset is caused by the acceleration or deceleration of the traveling equipment or the attitude change between the traveling equipment and the ground due to the unevenness of the road surface, so that such attitude change represents the relative attitude change between the traveling equipment and the ground, and may be a change in pitch attitude. The above phase offset information represents the offset amount of the corresponding vanishing points in the two images, and a part of this offset amount is caused by the horizontal running of the traveling equipment, and the other part is caused by the change in the pitch attitude of the traveling equipment. In some embodiments, the electronic device needs to determine the attitude change information of the traveling equipment based on the above phase offset information.

[0064] In some embodiments, the electronic device converts the above phase offset information according to a conversion relationship to obtain attitude change information of the traveling equipment. The attitude change information may refer to attitude change information in the vertical direction of the traveling equipment (also called pitch attitude change information).

[0065] In an embodiment of the present invention, the electronic device can predetermine reference attitude information, and the reference attitude information is determined based on at least two frames of historical road images. For example, the at least two frames of historical road images can be road images collected when the traveling device is traveling on a flat road surface at a constant or approximately constant speed, and the attitude information of the traveling device is determined based on the at least two frames of historical road images, and the attitude information is taken as reference attitude information (or also referred to as reference attitude information of the traveling device), and subsequently, the attitude change information obtained by the above technical solution of the present invention is all taken as a reference for the reference attitude information, and the attitude change information (which information can be a vector) is added to the reference attitude information to obtain the attitude information of the traveling device.

[0066] In some alternative embodiments of the invention, the method further comprises updating calibration information of the image acquisition component based on the attitude change information or the attitude information of the running equipment.

[0067] In an embodiment of the present invention, the electronic device can update calibration information of the image acquisition component based on the posture change information or the posture information of the traveling device, and the calibration information can be, for example, a homography matrix. The updated calibration information is used for bird's-eye view transformation, which includes bird's-eye view transformation from at least one of image information such as lanes, target objects, etc. to bird's-eye view information.

[0068] Based on the above embodiment, an embodiment of the present invention further provides an image processing device. Figure 4 is an exemplary structural diagram 1 of the configuration of an image processing device of an embodiment of the present invention. As shown in Figure 4, the device includes an acquisition unit 21, an attitude offset sensing unit 22 and an attitude determination unit 23, where: The acquisition unit 21 is configured to acquire at least two frames of road images through an image collection component provided in the running equipment, the attitude offset sensing unit 22 is configured to determine attitude change information of the running equipment between two frames of road images among the at least two frames of road images using a phase-only correlation method, and the attitude determination unit 23 is configured to determine attitude information of the running equipment based on the attitude change information and the reference attitude information of the running equipment.

[0069] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames, and to determine attitude change information of the traveling equipment between the road images of the two frames based on the phase offset information.

[0070] In some alternative embodiments of the present invention, the acquisition unit 21 is further configured to determine one or at least two regions of interest in the road image of each frame, at least one of the one or at least two regions of interest having at least one of the following characteristics: overlapping with the horizon and including a vanishing point.

[0071] In some alternative embodiments of the invention, there are no moving objects in the region of interest or the proportion of the moving objects occupying the region of interest is less than or equal to a first threshold.

[0072] In some alternative embodiments of the present invention, when the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest, where the first region of interest includes the vanishing point.

[0073] In some alternative embodiments of the present invention, the size of the second region of interest is smaller than the size of the first region of interest.

[0074] In some alternative embodiments of the present invention, the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps with the first region of interest.

[0075] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on a region of interest in the road image of each frame of the road images of the two frames.

[0076] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames when the first region of interest includes a moving object.

[0077] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine first phase offset information between the road images of the two frames based on the area of ​​interest in the road images of the two frames when a moving object is included in the area of ​​interest in each of the road images of the two frames, detect the proportion of the area in which the moving object is located in the area of ​​interest in each of the road images of the two frames, and determine phase offset information between the road images of the two frames based on the proportion and the first phase offset information.

[0078] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to determine second phase offset information between the road images of the two frames based on the regions of interest of each corresponding group in the road images of the two frames when the road image of each frame includes at least two regions of interest, where the positions of each region of interest of each group in each road image correspond, and to select one second phase offset information from the at least two second phase offset information as the phase offset information between the road images of the two frames, or to determine a median phase change information in the at least two second phase offset information and determine the median phase change information as the phase offset information between the road images of the two frames, or to determine an average value of the at least two second phase offset information and determine the average value as the phase offset information between the road images of the two frames.

[0079] In some alternative embodiments of the present invention, the at least one region of interest is symmetrical with respect to the horizon.

[0080] In some alternative embodiments of the present invention, the pixel points on each side of the at least one region of interest are 2 n , where n is a positive integer.

[0081] In some alternative embodiments of the present invention, the attitude offset sensing unit 22 is configured to: extract sub-images corresponding to regions of interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image, perform grayscale processing on the first sub-image and the second sub-image, respectively, obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image, perform Fourier transform processing on the first grayscale image and the second grayscale image, respectively, perform normalized cross processing on corresponding pixel points in the processed first grayscale image and the processed second grayscale image, obtain a processed image, perform inverse Fourier transform processing on the processed image, determine a peak position based on the value of each pixel point of the processed image, and determine phase offset information between the road images of the two frames based on the peak position.

[0082] In some alternative embodiments of the present invention, as shown in FIG. 5, the apparatus further comprises an update unit 24 configured to update calibration information of the image collection component based on the posture change information or posture information of the running equipment.

[0083] In an embodiment of the present invention, the device is applied to electronic equipment. In the device, the acquisition unit 21, the attitude offset sensing unit 22, the attitude determination unit 23 and the update unit 24 can all be realized by a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA) in practical application.

[0084] In addition, when the image processing device according to the above embodiment performs image processing, only the division of each of the above program modules is taken as an example for explanation, but in actual application, the above processing may be assigned to different program modules as necessary, that is, the internal structure of the device may be divided into different program modules to perform all or part of the above processing. Furthermore, the image processing device according to the above embodiment belongs to the same concept as the embodiment of the image processing method, and the implementation process thereof may be referred to the embodiment of the method, and will not be described again here.

[0085] An embodiment of the present invention further provides an electronic device, and FIG. 6 is an exemplary structural diagram of the hardware configuration of the electronic device of the embodiment of the present invention. As shown in FIG. 6, the electronic device comprises a memory 32, a processor 31, and a computer program stored in the memory 32 and executable by the processor 31, and the processor 31 realizes the steps of the image processing method described in the embodiment of the present invention by executing the program.

[0086] Optionally, the electronic device may further include a user interface 33 and a network interface 34. Here, the user interface 33 may include a display, a keyboard, a mouse, a trackball, a click wheel, a keystroke, a button, a touchpad or a touch screen, etc.

[0087] Optionally, each component in the electronic device is coupled via a bus system 35. It can be understood that the bus system 35 is used to realize the connection communication between these components. In addition to the data bus, the bus system 35 further includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the bus system 35 in FIG. 6 .

[0088] It can be understood that the memory 32 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Here, the non-volatile memory may be a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM), and the magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM) used as an external cache.By way of illustrative but non-limiting example, many forms of RAM are available, such as, for example, static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), SyncLink dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM), etc. Memory 32 as described in embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0089] The methods disclosed in the above embodiments of the present invention can be applied to or realized by the processor 31. The processor 31 can be an integrated circuit chip with signal processing capabilities. In the realization process, each step of the above methods can be performed by an integrated logic circuit of hardware in the processor 31 or an instruction in the form of software. The processor 31 can be a general-purpose processor, a DSP, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The processor 31 can realize or execute each method, step, and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor, or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly performed by a hardware decoding processor, or can be performed by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in the memory 32, and the processor 31 reads the information in the memory 32 and cooperates with the hardware to perform the steps of the above methods.

[0090] In an exemplary embodiment, the electronic device may be implemented with one or at least two Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, microprocessors, or other electronic elements to perform the above methods.

[0091] In an exemplary embodiment, the present invention further provides a computer readable storage medium, such as a memory 32, containing a computer program, which can be executed by a processor 31 of the electronic device to perform the steps of the above-mentioned method. The computer readable storage medium may be a memory, such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM, or may be a variety of devices including any of the above memories or any combination thereof.

[0092] A computer-readable storage medium according to an embodiment of the present invention stores a computer program for causing a processor to execute steps of an image processing method according to an embodiment of the present invention.

[0093] A computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device, and may be a volatile or non-volatile storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable CD-ROMs (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encoding devices (e.g., punch cards or grooved protruding structures on which instructions are stored), and any suitable combination thereof. A computer-readable storage medium as used herein is not to be construed as a momentary signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted by wires.

[0094] An embodiment of the present invention further provides a computer program, the computer program including computer readable code, which, when read and executed by a computer, implements some or all of the steps of the method in any one of the embodiments of the present invention.

[0095] The methods disclosed in the several method embodiments provided in the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0096] Features disclosed in the several product embodiments provided by the present invention may be combined in any non-conflicting manner to obtain new product embodiments.

[0097] The features disclosed in any method or apparatus embodiment provided in the present invention may be combined in any non-conflicting manner to obtain new method or apparatus embodiments.

[0098] In some embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be realized in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is merely a division of logical functions, and in actual implementation, there may be other division methods, for example, at least two units or components may be combined or integrated into another system, and some features may be ignored or not implemented. In addition, the coupling or direct coupling or communication connection between each component shown or discussed may be an indirect coupling or communication connection via some interfaces, devices or units, and may be in an electrical, mechanical, or other form.

[0099] The units described as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, may be located in one place or may be distributed across at least two network units, and some or all of the units therein may be selected according to actual needs to achieve the objective of the technical solutions in the embodiments of the present invention.

[0100] In addition, the functional units in each embodiment of the present invention may all be integrated into a single processing unit, or each unit may be separated into an independent unit, or two or more units may be integrated into a single unit, and the above integrated units may be realized as hardware or as a combination of hardware and software.

[0101] As can be understood by those skilled in the art, all or some of the steps of the above method embodiments can be performed by hardware associated with program instructions, and the program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are executed, and the storage medium includes a medium capable of storing program code, such as a mobile storage, a ROM, a RAM, a magnetic memory or an optical disk.

[0102] Alternatively, when the above-mentioned integrated units in the embodiments of the present invention are realized in the form of software functional modules and sold or used as an independent product, they can be stored in one computer-readable storage medium. Based on this understanding, the essential part of the technical solution of the present invention, i.e., the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product is stored in one storage medium and includes several instructions for causing one computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as mobile storage, ROM, RAM, magnetic disk, or optical disk.

[0103] The above content is merely an embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any modifications or replacements that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. 1. An image processing method, comprising: acquiring at least two frames of road images via an image acquisition component provided on the traveling equipment; determining attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images using a phase-only correlation method; determining posture information of the traveling equipment based on the posture change information and reference posture information of the traveling equipment.

2. Determining posture change information of the traveling equipment between two frames of road images among the at least two frames of road images using the phase-only correlation method includes: determining phase offset information between the road images of the two frames based on regions of interest in the road images of the two frames; and determining attitude change information of the traveling equipment between the road images of the two frames based on the phase offset information. The image processing method according to claim 1 .

3. The region of interest in the road image of each frame is by determining one or at least two regions of interest in the road image of each of the frames, at least one of the one or at least two regions of interest having at least one of the following characteristics: overlapping with a horizon line; and including a vanishing point. The image processing method according to claim 2 .

4. There are no moving objects in the region of interest, or a proportion of the moving objects occupying the region of interest is less than or equal to a first threshold; 4. The image processing method according to claim 2.

5. When the road image of each frame includes at least two regions of interest, the at least two regions of interest include at least a first region of interest and a second region of interest; The first region of interest includes the vanishing point.

5. The image processing method according to claim 3.

6. The size of the second region of interest is smaller than the size of the first region of interest. The image processing method according to claim 5.

7. the second region of interest is within the first region of interest, or the second region of interest is outside the first region of interest, or the second region of interest partially overlaps the first region of interest.

7. The image processing method according to claim 5 or 6.

8. Determining phase offset information between the road images of the two frames based on a region of interest in the road images of the two frames includes: determining phase offset information between the road images of the two frames based on a region of interest in the road image of each frame of the road images of the two frames; 7. The image processing method according to claim 4.

9. The first region of interest includes a moving object, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes: determining phase offset information between the road images of the two frames based on a second region of interest in the road images of the two frames; 7. The image processing method according to claim 5 or 6.

10. A region of interest in the road image of each of the two frames of the road images includes a moving object, and determining phase offset information between the road images of the two frames based on the region of interest in the road images of the two frames includes: determining a first phase offset information between the road images of the two frames based on a region of interest in the road images of the two frames; detecting a ratio of an area in which the moving object is located in a region of interest in each of the road images of the two frames, and determining phase offset information between the road images of the two frames based on the ratio and the first phase offset information; 4. The image processing method according to claim 2.

11. The road image of each frame includes at least two regions of interest, and determining phase offset information between the road images of the two frames based on the regions of interest in the road images of the two frames includes: determining second phase offset information between the road images of the two frames based on the corresponding groups of regions of interest in the road images of the two frames, where the positions of the regions of interest in each group in the respective road images correspond to each other; selecting one second phase offset information from at least two pieces of second phase offset information as the phase offset information between the road images of the two frames, or determining a median phase change information in the at least two pieces of second phase offset information and determining the median phase change information as the phase offset information between the road images of the two frames, or determining an average value of the at least two pieces of second phase offset information and determining the average value as the phase offset information between the road images of the two frames. The image processing method according to claim 2 .

12. the at least one region of interest is symmetrical with respect to the horizon; The image processing method according to any one of claims 2 to 11.

13. The pixel points of each side of the at least one region of interest are 2 to the power of 2, where n is a positive integer. The image processing method according to any one of claims 2 to 11.

14. Determining phase offset information between the road images of the two frames based on a region of interest in the road images of the two frames includes: Extracting sub-images corresponding to an area of ​​interest in the road images of the two frames, respectively, to obtain a first sub-image and a second sub-image; performing grayscale processing on the first sub-image and the second sub-image, respectively, to obtain a first grayscale image corresponding to the first sub-image and a second grayscale image corresponding to the second sub-image; performing a Fourier transform process on the first grayscale image and the second grayscale image, respectively, and performing a normalized cross process on corresponding pixel points in the processed first grayscale image and the processed second grayscale image to obtain a processed image; performing an inverse Fourier transform process on the processed image, determining a peak position based on the value of each pixel point of the processed image, and determining phase offset information between the road images of the two frames based on the peak positions. The image processing method according to claim 2 .

15. The image processing method includes: Further comprising updating calibration information of the image acquisition component based on the attitude change information or the attitude information of the traveling equipment.

15. The image processing method according to claim 1.

16. An image processing apparatus comprising: an acquisition unit; an attitude offset sensing unit; and an attitude determination unit; The acquisition unit is configured to acquire at least two frames of road images via an image acquisition component provided on a traveling device; The attitude offset sensing unit is configured to determine attitude change information of the traveling equipment between two frames of road images among the at least two frames of road images by using a phase-only correlation method; The image processing device, wherein the attitude determination unit is configured to determine attitude information of the traveling equipment based on the attitude change information and reference attitude information of the traveling equipment.

17. A computer readable storage medium having stored thereon a computer program for causing a processor to carry out the steps of the image processing method according to any one of claims 1 to 15.

18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the processor executing the computer program to realize the steps of the image processing method according to any one of claims 1 to 15.

19. A computer program comprising computer readable code for causing a processor of an equipment to carry out the steps of the image processing method according to any one of claims 1 to 15.

20. A computer program product having stored thereon computer readable instructions for causing a computer to carry out the steps of the image processing method according to any one of claims 1 to 15.

Images