Multi-camera-based surround-view stereo image acquisition device and method

Abstract

The invention discloses a multi-view camera-based surround-view stereo image acquisition device and method, and the method comprises the following steps: an image acquisition card acquires original images of four wide-angle cameras to obtain four original images P1, P2, P3 and P4, and stores the four original images in a cache; the original image is divided into a left image and a right image in the middle, and four PiL original images and four PiR original images are obtained respectively; splicing the four PiLs into a left side look-around view PL of a left side view angle; splicing the four original images PiR into a right-side look-around image PR of a right-side view angle; and processing the left side look-around image PL and the right side look-around image PR by adopting a binocular stereo matching principle to obtain a 360-degree look-around stereo image PD. Compared with a common panoramic camera or a binocular stereo camera, the total field angle gt of image acquisition can be met; in other words, two 360-degree panoramic images with different visual angles are generated and then stereo matching is carried out, so that the problem that a common panoramic camera cannot effectively form stereo perception is solved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a surround-view stereo image acquisition device and method based on multi-cameras. Background Art

[0002] Visual perception technology is crucial to robots, enabling them to perceive the environment, make decisions and perform tasks. Robots use visual sensors to collect images and achieve functions such as autonomous navigation and obstacle avoidance.

[0003] The binocular stereo vision sensors widely used in the field of machine vision have some limitations. Although this sensor can simulate human binocular vision and calculate the distance and three-dimensional shape of objects through image differences, its field of view is small, resulting in the need for multiple sensors to cover different directions, which increases cost and volume, and also makes information fusion and matching difficult.

[0004] In addition, although the panoramic cameras used by robots can provide a 360-degree view, they usually cannot generate stereo images and lack depth information, which limits their use in application scenarios that require accurate depth perception, such as robot navigation and precision measurement. This challenge is also a difficult problem that needs to be overcome in the current technological development.

[0005] Therefore, developing a stereo vision sensor that can achieve 360-degree omnidirectional coverage and a surround stereo image acquisition method has become a technical problem that needs to be urgently solved in this field.

[0006] Through the above analysis, the problems and defects of the prior art are as follows:

[0007] Existing binocular stereo vision sensors simulate the binocular vision of the human eye and calculate the distance and three-dimensional shape of an object by comparing the difference between two images. Their field of view is small, which limits the robot's perception, operation and safe operation capabilities. Using multiple sets of binocular stereo vision sensors for multi-angle coverage brings about the problems of high cost, large weight and volume, and difficulty in fusing and matching the information collected by different sensors. Ordinary panoramic cameras can only generate two-dimensional images and cannot effectively form three-dimensional perception.

[0008] The significance of solving the above technical problems is:

[0009] The present invention uses multiple wide-angle cameras to synchronously collect and process the images, decomposes and splices them into surround images from left and right perspectives, and uses binocular stereo imaging matching to achieve the effect of collecting 360-degree surround stereo images without blind spots outside the robot. At the same time, it effectively reduces the cost, volume and weight, and provides global synchronous surround stereo image collection, effectively improving the external perception and intelligent operation capabilities of robot applications. Summary of the invention

[0010] In order to overcome the problems existing in the related art, the disclosed embodiment of the present invention provides a surround stereo image acquisition device and method based on a multi-camera, and the technical solution is as follows:

[0011] A surround stereo image acquisition device based on multi-cameras comprises a robot body, on which four wide-angle cameras with a field of view greater than 180 degrees are equally fixed along the circumference, the wide-angle cameras are connected to an image acquisition card via image signal lines, and the exposure control pins of the four wide-angle cameras are simultaneously connected to the image acquisition card via the same shutter signal line.

[0012] Furthermore, the four wide-angle cameras are arranged at an angle of 90 degrees to each other and are synchronously controlled through the same shutter signal line.

[0013] Another object of the present invention is to provide a surround-view stereo image acquisition method based on a multi-camera, using a surround-view stereo image acquisition device based on a multi-camera, comprising the following steps:

[0014] S1: The image acquisition card sends a pulse signal through the shutter signal line to control the four wide-angle cameras to expose at the same time. The image acquisition card collects the original images of the four wide-angle cameras, obtains four original images P1, P2, P3, and P4, and stores them in the cache;

[0015] S2: Split the four original images P1, P2, P3, and P4 obtained in step S1 into two images, one on the left and one on the right, to obtain four PiL original images and four PiR original images, where i is the wide-angle camera number;

[0016] S3: stitching the four original images PiL obtained in step S2 into a left side surround view PL of the left side perspective;

[0017] S4: stitching the four original images PiR obtained in step S2 into a right side surround view PR of the right perspective;

[0018] S5: The left surround view PL and the right surround view PR obtained in step S3 and step S4 are processed using the binocular stereo matching principle to obtain a 360-degree surround stereo image PD.

[0019] Furthermore, in step S3 and step S4, the mapping relationship between each image coordinate system in the four wide-angle cameras is calculated by using perspective transformation and inverse perspective transformation methods, and the original image PiL is mapped to a common left surround view PL, and the original image PiR is mapped to a common right surround view PR.

[0020] Further, mapping to a common surround image comprises the following steps:

[0021] S501: Obtain the internal and external parameters of each wide-angle camera, which are defined as Ki and [Ri|ti] respectively; S502: Convert each image coordinate (ui,vi) into a surround view coordinate system using the following formula;

[0022] (u,v)=f(ui,vi,Ki,[Ri|ti]) (1)

[0023] Where: f is the perspective transformation function;

[0024] S503: The four images converted into the surround view coordinate system are stitched together into a surround view image, and finally a left surround view PL and a right surround view PR are obtained.

[0025] Further, in step S5, feature points are extracted from the left surround view PL and the right surround view PR to identify key points in the image, including corner points, edges or other significant features, and then corresponding points in the left surround view PL and the right surround view PR are found;

[0026] According to the disparity and the baseline distance, that is, the horizontal displacement between the corresponding points of the left surround view PL and the right surround view PR, and the distance between the geometric centers of the left surround view PL and the right surround view PR, the depth of the corresponding point is calculated, and the depth map PD is further obtained. The calculation formula is as follows:

[0027] PD(u, v) = B × f / d(u, v) (2)

[0028] in:

[0029] B is the baseline distance; f is the focal length of the camera; d(u, v) is the disparity, i.e., the horizontal pixel difference between the left surround view PL and the right surround view PR for the same scene point;

[0030] Through the above method, the left surround view PL and the right surround view PR are combined to generate a comprehensive surround stereo image PD.

[0031] Further, the disparity map obtained by horizontal displacement between corresponding points from the left surround view PL to the right surround view PR is optimized by applying median filtering, bilateral filtering or disparity map refinement to eliminate discontinuities and inaccurate areas in the disparity map.

[0032] Another object of the present invention is to provide an information data processing terminal, which is used to implement a surround stereo image acquisition method based on multiple cameras.

[0033] Another object of the present invention is to provide a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the steps of a surround-view stereo image acquisition method based on multiple cameras.

[0034] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a surround stereo image acquisition method based on multiple cameras are implemented.

[0035] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows:

[0036] Wide-angle cameras arranged in four directions of the robot are used to synchronously collect surrounding images and perform algorithm processing, which are decomposed and spliced ​​into surround view images from left and right perspectives. The binocular stereo imaging matching method is used to achieve the technical effect of collecting 360-degree surround stereo image perception of the robot's exterior without blind spots.

[0037] Compared with ordinary panoramic cameras or binocular stereo cameras, the present invention can meet the requirement of a total field of view of >720 degrees, that is, generate two 360-degree surround images with different viewing angles and then perform stereo matching. At the same time, it solves the problem that multiple binocular stereo vision sensors are expensive, heavy and bulky, and the information collected by different sensors is difficult to merge and match, and ordinary panoramic cameras cannot effectively form stereo perception.

[0038] At the same time, the cost of the robot's binocular stereo vision sensor system can be further reduced, improving operational capabilities and safety. This cost saving makes robots economically feasible in more industries and application scenarios, thereby expanding market share. It improves the commercial value and competitiveness of the robot system and has high commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0040] Figure 1 It is a connection diagram of a surround-view stereo image acquisition device based on a multi-camera provided in an embodiment of the present invention;

[0041] Figure 2 It is a flow diagram of a surround-view stereo image acquisition method based on multi-cameras provided in an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of generating a surround stereoscopic image provided by an embodiment of the present invention;

[0043] Figure 4is a schematic diagram of the stereoscopic field of view perception range achieved by an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the field of view of a traditional binocular stereo vision sensor;

[0045] Figure 6 is a schematic diagram of the perception field of view implemented by an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of the perception field of view of a traditional binocular stereo vision sensor. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0048] Embodiment 1:

[0049] like Figure 1 As shown, a surround stereo image acquisition device based on multi-cameras includes a robot body, on which four wide-angle cameras with a field of view greater than 180 degrees are equally divided and fixed along the circumference, the wide-angle cameras are connected to an image acquisition card via image signal lines, and the exposure control pins of the four wide-angle cameras are simultaneously connected to the image acquisition card via the same shutter signal line.

[0050] Furthermore, in the embodiment, it can also be considered that the four wide-angle cameras are arranged at an angle of 90 degrees to each other and are synchronously controlled through the same shutter signal line.

[0051] Directions:

[0052] like Figure 1 As shown, wide-angle cameras are fixed on four directions of the robot body, and are placed at 90 degrees to each other. The field of view of each wide-angle camera is a wide-angle camera of >180 degrees.

[0053] Use an image signal line to connect each camera to the image acquisition card to collect image information. Use the same shutter signal line to connect to the exposure control pin of each camera to control all cameras to expose and take pictures at the same time.

[0054] Embodiment 2:

[0055] like Figure 2-Figure 3As shown, a surround-view stereo image acquisition method based on a multi-camera, using a surround-view stereo image acquisition device based on a multi-camera, includes the following steps:

[0056] S1: The image acquisition card sends a pulse signal through the shutter signal line to control the four wide-angle cameras to expose at the same time. The image acquisition card collects the original images of the four wide-angle cameras, obtains four original images P1, P2, P3, and P4, and stores them in the cache;

[0057] S2: Split the four original images P1, P2, P3, and P4 obtained in step S1 into two images, one on the left and one on the right, to obtain four PiL original images and four PiR original images, where i is the wide-angle camera number;

[0058] S3: stitching the four original images PiL obtained in step S2 into a left side surround view PL of the left side perspective;

[0059] S4: stitching the four original images PiR obtained in step S2 into a right side surround view PR of the right perspective;

[0060] S5: The left surround view PL and the right surround view PR obtained in step S3 and step S4 are processed using the binocular stereo matching principle to obtain a 360-degree surround stereo image PD.

[0061] In this embodiment, four original images P1, P2, P3, and P4 are first collected by four wide-angle cameras, and then they are respectively split in the center to obtain four PiL original images and four PiR original images, where i is the wide-angle camera number. According to this numbering rule, eight original images P1L, P1R, P2L, P2R, P3L, P3R, P4L, and P4R are obtained, and then the original images P1L, P2L, P3L, and P4L are spliced ​​into a left-side surround view PL from a left-side perspective, and the original images P1R, P2R, P3R, and P4R are spliced ​​into a right-side surround view PR from a right-side perspective.

[0062] At this time, two surround views PL and PR with left-right parallax characteristics are obtained. Based on the binocular stereo matching principle, a 360-degree surround stereo image PD can be obtained, which is the ultimate target result of the present invention.

[0063] Furthermore, in the embodiment, it can also be considered that in step S3 and step S4, the mapping relationship between each image coordinate system in the four wide-angle cameras is calculated by using the perspective transformation and inverse perspective transformation methods, and the original image PiL is mapped to the common left-side surround view PL, and the original image PiR is mapped to the common right-side surround view PR.

[0064] In this embodiment, the original images P1L, P2L, P3L, and P4L are fused into a complete image in a plane coordinate system, and a perspective transformation and an inverse perspective transformation method are required to calculate the mapping relationship between different image coordinate systems.

[0065] According to the multi-camera geometric imaging model, each image PiL needs to be mapped to a common left surround view PL.

[0066] The original images P1R, P2R, P3R, and P4R are also mapped to a common right-side surround view PR by using perspective transformation and inverse perspective transformation methods.

[0067] Furthermore, in an embodiment, it may be considered that mapping to a common surround view image includes the following steps:

[0068] S501: Obtain the internal and external parameters of each wide-angle camera, which are defined as Ki and [Ri|ti] respectively; S502: Convert each image coordinate (ui,vi) into a surround view coordinate system using the following formula;

[0069] (u,v)=f(ui,vi,Ki,[Ri|ti]) (1)

[0070] Where: f is the perspective transformation function;

[0071] S503: The four images converted into the surround view coordinate system are stitched together into a surround view image, and finally a left surround view PL and a right surround view PR are obtained.

[0072] Furthermore, in the embodiment, it can also be considered that in step S5, feature points are extracted from the left surround view PL and the right surround view PR, key points in the image are identified, including corner points, edges or other significant features, and then corresponding points in the left surround view PL and the right surround view PR are found;

[0073] According to the disparity and the baseline distance, that is, the horizontal displacement between the corresponding points of the left surround view PL and the right surround view PR, and the distance between the geometric centers of the left surround view PL and the right surround view PR, the depth of the corresponding point is calculated, and the depth map PD is further obtained. The calculation formula is as follows:

[0074] PD(u, v) = B × f / d(u, v) (2)

[0075] in:

[0076] B is the baseline distance; f is the focal length of the camera; d(u, v) is the disparity, i.e., the horizontal pixel difference between the left surround view PL and the right surround view PR for the same scene point;

[0077] Through the above method, the left surround view PL and the right surround view PR are combined to generate a comprehensive surround stereo image PD.

[0078] In this embodiment, the key points in the image give priority to corner points and edges, or other significant features are selected according to the implementation situation, including but not limited to features such as image structure, texture, color and geometric information.

[0079] Furthermore, in the embodiments, it can also be considered that the disparity map obtained by the horizontal displacement between the corresponding points of the left surround view PL and the right surround view PR is optimized by applying median filtering, bilateral filtering or disparity map refinement to eliminate discontinuities and inaccurate areas in the disparity map.

[0080] In this embodiment, median filtering, bilateral filtering or disparity map refinement are all existing technologies, and are combined with the method of the present invention to further improve the image optimization quality.

[0081] Application Example 1:

[0082] To further illustrate the effects of the embodiments of the present invention, the following experiments are performed:

[0083] As shown in the figure, Figure 4 is the range of the stereoscopic field of view perceived by the device of the present invention, Figure 5 It is the field of view perceived by ordinary traditional binocular stereo vision sensors.

[0084] The robot using the device and method can autonomously perceive the 360-degree stereoscopic environment information around it and return photos such as Figure 6 At this point, the robot can effectively identify obstacles in front, behind, left, and right, and choose a safe route in any direction.

[0085] Figure 7 For robots that use ordinary binocular stereo vision sensors, they can only detect obstacles in the front direction, and their field of view is limited, making it impossible to select a better route in other directions.

[0086] By comparison, it can be seen that the robot system using the device of the present invention has a much larger field of view than the robot using an ordinary binocular stereo vision sensor.

[0087] Since the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention, their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0088] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment.

[0089] An embodiment of the present invention further provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0090] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0091] An embodiment of the present invention further provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, and switches.

[0092] An embodiment of the present invention further provides a server, which is used to provide a user input interface to implement the steps in the above method embodiments when executed on an electronic device.

[0093] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0095] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A surround stereo image acquisition device based on a multi-camera, comprising a robot body, characterized in that: The robot body is equally divided along the circumference and fixed with four wide-angle cameras with a field of view greater than 180 degrees. The wide-angle cameras are connected to an image acquisition card via image signal lines, and the exposure control pins of the four wide-angle cameras are simultaneously connected to the image acquisition card via the same shutter signal line.

2. The surround stereo image acquisition device based on multi-camera according to claim 1, characterized in that: The four wide-angle cameras are arranged at an angle of 90 degrees to each other and are synchronously controlled through the same shutter signal line.

3. A surround-view stereo image acquisition method based on a multi-camera, using the surround-view stereo image acquisition device based on a multi-camera according to any one of claims 1 to X, characterized in that: The following steps are involved: S1: The image acquisition card sends a pulse signal through the shutter signal line to control the four wide-angle cameras to expose at the same time. The image acquisition card collects the original images of the four wide-angle cameras, obtains four original images P1, P2, P3, and P4, and stores them in the cache; S2: Split the four original images P1, P2, P3, and P4 obtained in step S1 into two images, one on the left and one on the right, to obtain four PiL original images and four PiR original images, where i is the wide-angle camera number; S3: stitching the four original images PiL obtained in step S2 into a left side surround view PL of the left side perspective; S4: stitching the four original images PiR obtained in step S2 into a right side surround view PR of the right perspective; S5: The left surround view PL and the right surround view PR obtained in step S3 and step S4 are processed using the binocular stereo matching principle to obtain a 360-degree surround stereo image PD.

4. The surround stereo image acquisition method based on multi-camera according to claim 3, characterized in that: In step S3 and step S4, the mapping relationship between each image coordinate system in the four wide-angle cameras is calculated by using perspective transformation and inverse perspective transformation methods, and the original image PiL is mapped to a common left surround view PL, and the original image PiR is mapped to a common right surround view PR.

5. The surround stereo image acquisition method based on multi-cameras according to claim 4 is characterized in that: Mapping to a common surround image involves the following steps: S501: Obtain the internal and external parameters of each wide-angle camera, which are defined as Ki and [Ri|ti] respectively; S502: Convert each image coordinate (ui,vi) into a surround view coordinate system using the following formula; (u,v)=f(ui,vi,Ki,[Ri|ti]) (1) Where: f is the perspective transformation function; S503: The four images converted into the surround view coordinate system are stitched together into a surround view image, and finally a left surround view PL and a right surround view PR are obtained.

6. The surround stereo image acquisition method based on multi-cameras according to claim 5, characterized in that: In step S5, feature points are extracted from the left surround view PL and the right surround view PR to identify key points in the image, including corner points, edges or other significant features, and then corresponding points in the left surround view PL and the right surround view PR are found; According to the disparity and the baseline distance, that is, the horizontal displacement between the corresponding points of the left surround view PL and the right surround view PR, and the distance between the geometric centers of the left surround view PL and the right surround view PR, the depth of the corresponding point is calculated, and the depth map PD is further obtained. The calculation formula is as follows: PD(u, v) = B × f / d(u, v) (2) in: B is the baseline distance; f is the focal length of the camera; d(u, v) is the disparity, i.e., the horizontal pixel difference between the left surround view PL and the right surround view PR for the same scene point; Through the above method, the left surround view PL and the right surround view PR are combined to generate a comprehensive surround stereo image PD.

7. The method for acquiring surround stereo images based on multiple cameras according to claim 6, characterized in that: The disparity map obtained by horizontal displacement between corresponding points of the left surround view PL and the right surround view PR is optimized by applying median filtering, bilateral filtering or disparity map refinement to eliminate discontinuities and inaccurate areas in the disparity map.

8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the surround stereo image acquisition method based on multiple cameras as described in any one of claims 3-7.

9. A computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the steps of the surround stereo image acquisition method based on multi-cameras as described in any one of claims 3 to 7.

10. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the surround stereo image acquisition method based on multi-cameras as described in any one of claims 3 to 7 when executing the computer program.

Images