A real-time data processing method for three-eye camera for multi-view image acquisition

Abstract

The present invention relates to the field of computer vision technology, and provides a three-eye camera real-time data processing method, system, medium and electronic device for multi-view image acquisition. The image data of the target scene is collected by a left camera, a middle camera and a right camera, and the image data collected by the left camera, the middle camera and the right camera are subjected to a first distortion correction and a second distortion correction; feature point detection is performed based on the corrected image data, and a matching relationship between the image data is established; the basic matrix and the intrinsic matrix of the image data are calculated according to the matching relationship, and the depth information of the target scene is obtained by using triangulation; the depth information is fused with the corrected image data to generate three-dimensional point cloud data with depth annotation. Not only the quality of the image data is improved, but also the accuracy of the depth information is enhanced, with higher depth estimation accuracy and robustness, and it is suitable for scenes with a variety of depth ranges.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, computer-readable medium and electronic device for real-time data processing of a three-eye camera for multi-view image acquisition. Background Art

[0002] With the vigorous development of computer vision technology, stereo vision systems based on multi-cameras have shown broad application prospects in the fields of robot vision, autonomous driving, and three-dimensional reconstruction. However, due to the inevitable optical distortion phenomena such as radial distortion and tangential distortion in the camera imaging process, the acquired images are distorted, which seriously affects the subsequent feature matching and depth estimation accuracy. Traditional distortion correction methods are mainly based on polynomial models. This method often shows large correction errors when processing images with large field of view and strong distortion. At the same time, with the continuous improvement of camera resolution, the computational complexity of traditional correction algorithms has increased dramatically, making it difficult to meet the needs of real-time processing.

[0003] The distortion correction technology currently used in the industry has many limitations: first, the traditional single distortion correction model is difficult to handle radial distortion and tangential distortion at the same time, resulting in unsatisfactory correction effects; second, most existing correction algorithms use global optimization strategies, which are computationally intensive and time-consuming, and cannot meet real-time processing requirements; third, in multi-camera systems, due to differences in distortion parameters of each camera, traditional unified correction methods are difficult to ensure geometric consistency between multiple perspective images, which directly affects the accuracy of subsequent stereo matching. In addition, existing technologies lack adaptive capabilities when dealing with different degrees of distortion, and the correction effect often depends on the adjustment of artificial experience parameters. Summary of the invention

[0004] Embodiments of the present invention provide a three-eye camera real-time data processing system, method, computer-readable medium and electronic device for multi-view image acquisition, thereby at least to a certain extent solving the problem of difficulty in ensuring geometric consistency between multiple view images.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to one aspect of the present invention, a real-time data processing method of a three-eye camera for multi-view image acquisition is provided, comprising:

[0007] The image data of the target scene is collected by the left camera, the middle camera and the right camera.

[0008] Performing a first distortion correction and a second distortion correction on the image data collected by the left camera, the middle camera and the right camera;

[0009] Perform feature point detection based on the corrected image data and establish a matching relationship between the image data;

[0010] Calculating a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtaining depth information of the target scene by using a triangulation method;

[0011] The depth information is fused with the corrected image data to generate three-dimensional point cloud data with depth annotations.

[0012] In the present invention, based on the above-mentioned solution, the left camera and the middle camera are fixedly installed on the same horizontal plane and maintain a first preset distance; the middle camera and the right camera are fixedly installed on the same horizontal plane and maintain a second preset distance;

[0013] The optical axes of the left camera, the middle camera and the right camera are arranged in parallel and perpendicular to the same horizontal plane, and the left camera, the middle camera and the right camera synchronously collect image data of the target scene through a synchronous trigger signal.

[0014] In the present invention, based on the above solution, the first distortion correction includes:

[0015] Determine the optical center position of the image, and then calculate the distance from each pixel to the optical center; based on the distance from each pixel to the optical center, use a polynomial model to model the distortion; at the same time, different radial distortion coefficients control the power terms of the polynomial model, and work together to correct the distortion effect;

[0016] The second distortion correction comprises:

[0017] The result of the first distortion correction is further optimized to correct the degree of lens tilt at different temperatures.

[0018] In the present invention, based on the above solution, the first distortion correction further includes adaptive weight and Gaussian attenuation terms;

[0019] The adaptive weight dynamically adjusts the correction strength by calculating the depth estimation error of the feature point;

[0020] The Gaussian attenuation term allows the correction effect to transition smoothly as the distance increases, thereby avoiding sudden changes or unnaturalness in the edge areas of the image during the correction process.

[0021] In the present invention, based on the above-mentioned solution, a radial distortion correction matrix and a tangential distortion correction matrix are constructed based on the results of the first distortion correction and the second distortion correction;

[0022] Applying the radial distortion correction matrix and the tangential distortion correction matrix in cascade to the original image data to obtain corrected image data;

[0023] The radial distortion correction matrix is ​​shown below:

[0024]

[0025] in, and are the focal length parameters of the camera in the x and y directions, is the optical center coordinate, is the square of the distance from the pixel to the optical center, is the radial distortion coefficient, is the adaptive weight coefficient, is the Gaussian attenuation factor;

[0026] The tangential distortion correction matrix is ​​shown below:

[0027]

[0028] in, is the first tangential distortion coefficient, It mainly describes the cross distortion in the xy plane. It is the second tangential distortion coefficient, which describes the cross distortion in another direction. is the original image coordinate, is the tilt coefficient after temperature compensation, is the current temperature, is the temperature compensation function.

[0029] In the present invention, based on the aforementioned solution, the feature point detection includes constructing a scale space pyramid for the corrected left camera image, the middle camera image and the right camera image, respectively, and performing feature point detection at each scale space layer;

[0030] The feature point detection uses the Hessian matrix, as shown in the following formula:

[0031]

[0032] in, , , is the second-order derivative of the corrected image in the corresponding direction, is the adaptive weight matrix based on image content:

[0033]

[0034] in, is the image gradient amplitude, and To adjust the parameters.

[0035] In the present invention, based on the above scheme, the matching relationship between the image data is established by using a hierarchical matching strategy; the hierarchical matching strategy includes coarse matching and fine matching;

[0036] The similarity metric of the rough matching stage The calculation of is as follows:

[0037]

[0038] in, and is the descriptor of the feature point to be matched, is the spatial distance of feature points, is the Gaussian kernel parameter of the distance weight, and represents the feature points to be matched;

[0039] The matching confidence of the fine matching stage The calculation of is as follows:

[0040]

[0041] in, is the three-view geometric error, is the scale factor, is the coarse matching similarity between points p and q, is the coarse matching similarity between points q and r.

[0042] According to one aspect of the present invention, a three-eye camera real-time data processing system for multi-view image acquisition is provided, comprising:

[0043] The image acquisition unit collects image data of the target scene through the left camera, the middle camera and the right camera.

[0044] A correction unit, performing a first distortion correction and a second distortion correction on the image data collected by the left camera, the middle camera and the right camera;

[0045] A feature detection unit, performing feature point detection based on the corrected image data to establish a matching relationship between the image data;

[0046] A feature processing unit calculates a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtains depth information of the target scene by using a triangulation method;

[0047] A fusion unit is used to fuse the depth information with the corrected image data to generate three-dimensional point cloud data with depth annotations.

[0048] According to one aspect of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the real-time data processing method of a three-eye camera for multi-view image acquisition as described in the above embodiment is implemented.

[0049] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time data processing method of a three-eye camera for multi-view image acquisition as described in the above-mentioned embodiments.

[0050] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the three-eye camera real-time data processing method for multi-view image acquisition provided in the above-mentioned various optional implementations.

[0051] In the technical solution of the present invention, the target scene image is collected and distortion correction is performed through a multi-view camera group, and the depth information of the target scene is accurately calculated in combination with the improved SURF feature point detection and matching method. The depth estimation accuracy in different depth ranges is optimized through an asymmetric camera layout. The combination of double-layer distortion correction, feature point matching and triangulation not only improves the quality of image data, but also enhances the accuracy of depth information, and finally generates three-dimensional point cloud data with depth annotations, which has higher depth estimation accuracy and robustness and is suitable for scenes with a variety of depth ranges.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings herein are incorporated into and constitute a part of the specification, showing embodiments consistent with the present invention, and together with the specification, are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0054] Figure 1 The present invention schematically shows a flow chart of a method for real-time data processing of a three-eye camera for multi-view image acquisition in an embodiment of the present invention.

[0055] Figure 2 A flowchart of a multi-layer correction scheme in one embodiment of the present invention is schematically shown.

[0056] Figure 3 A schematic diagram of a three-eye camera real-time data processing system for multi-view image acquisition in one embodiment of the present invention is schematically shown.

[0057] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0059] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0060] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0061] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0062] The implementation details of the technical solution of the present invention are described in detail below:

[0063] Figure 1 A flow chart of a method for real-time data processing of a three-eye camera for multi-view image acquisition according to an embodiment of the present invention is shown. Figure 1As shown, the three-eye camera real-time data processing method for multi-view image acquisition includes at least steps S110 to S150, which are described in detail as follows:

[0064] S110: Collect image data of the target scene through the left camera, the middle camera and the right camera.

[0065] Multi-view images of a target scene are captured by a three-eye camera group with a preset spacing. Specifically, the three-eye camera group includes a left camera, a middle camera and a right camera. The left camera and the middle camera are fixedly installed on the same horizontal plane and maintain a first preset spacing, and the first preset spacing is 6 cm. The middle camera and the right camera are fixedly installed on the same horizontal plane and maintain a second preset spacing, and the second preset spacing is 10 cm. The optical axes of the left camera, the middle camera and the right camera are arranged in parallel and perpendicular to the same horizontal plane. The left camera, the middle camera and the right camera synchronously capture image data of the target scene through a synchronous trigger signal, wherein the target scene is located within the common field of view of the three cameras, and the depth of the field of view is between 0.5 meters and 5 meters.

[0066] Among them, the setting mode in which the first preset distance is smaller than the second preset distance makes the binocular system composed of the left camera and the middle camera suitable for accurate depth estimation of close-range targets, and the binocular system composed of the middle camera and the right camera suitable for reliable depth estimation of long-range targets. Through this asymmetric layout, the depth estimation accuracy of the system in different depth ranges is improved.

[0067] S120: Perform a first distortion correction and a second distortion correction on the image data captured by the left camera, the middle camera and the right camera.

[0068] like Figure 2 As shown, in one embodiment of the present invention, a double-layer distortion correction process is performed on the image data collected by the left camera, the middle camera and the right camera, and the double-layer distortion correction process includes a first distortion correction and a second distortion correction, wherein the first distortion correction uses an improved five-parameter radial distortion model to correct the image data, and the correction equation of the improved five-parameter radial distortion model is:

[0069]

[0070]

[0071] in, is the original image coordinate, ( ) is the first corrected coordinate, is the square of the distance from the pixel to the optical center, is the optical center coordinate, is the radial distortion coefficient, and the newly added is the adaptive weight coefficient, is the Gaussian attenuation factor, the adaptive weight coefficient Calculated by the following formula:

[0072]

[0073] in, is the adjustment coefficient, and its value range is [0.1, 0.3]. For the The depth estimation error of feature points is is the total number of feature points.

[0074] The second distortion correction uses a three-parameter tangential distortion model that takes temperature influence into consideration to further correct the image data after the first distortion correction. The improved tangential distortion correction equation is:

[0075]

[0076]

[0077]

[0078] in, is the second correction coordinate, , is the tangential distortion coefficient, is the tilt coefficient after temperature compensation, is the reference tilt coefficient, is the temperature change relative to the calibration temperature, and are the tangential distortion temperature coefficient and the tilt temperature coefficient respectively; the optimization objective function of the tangential distortion correction matrix obtained by iteratively optimizing the tangential distortion coefficient and the tilt coefficient by the least squares method is:

[0079]

[0080] in, is the feature point weight, and are corresponding point pairs, is the temperature-dependent regularization coefficient;

[0081] Then, based on the above radial distortion correction equation, the radial distortion correction matrix is ​​constructed: :

[0082]

[0083] in, and are the focal length parameters of the camera in the x and y directions respectively. This matrix converts the nonlinear correction of radial distortion into a linear transformation in homogeneous coordinates;

[0084] Based on the tangential distortion correction equation, a temperature-adaptive tangential distortion correction matrix is ​​constructed. :

[0085]

[0086] in, is the first tangential distortion coefficient, It mainly describes the cross distortion in the xy plane. It is the second tangential distortion coefficient, which describes the cross distortion in another direction. is the original image coordinate, is the tilt coefficient after temperature compensation, is the current temperature, is the temperature compensation function.

[0087] The cascade effect of the two correction matrices is achieved through matrix multiplication to obtain a comprehensive correction matrix. The radial distortion correction matrix and the tangential distortion correction matrix are cascaded to the original image data to obtain corrected image data, and the corrected image data satisfies the epipolar constraint condition.

[0088] S130: Perform SURF feature point detection based on the corrected image data to establish a matching relationship between the image data;

[0089] In one embodiment of the present invention, improved SURF feature point detection and matching is performed based on the corrected image data, specifically including:

[0090] First, a scale space pyramid is constructed for the corrected left camera image, middle camera image and right camera image respectively, and the scale space pyramid adopts an adaptive sampling interval. Secondly, an improved Hessian matrix is ​​used to detect feature points at each scale space layer, and the improved Hessian matrix is ​​defined as:

[0091]

[0092] in, , , is the second-order derivative of the corrected image in the corresponding direction, is the adaptive weight matrix based on image content:

[0093]

[0094] In the formula, is the image gradient amplitude, and To adjust the parameters; then, extract the improved SURF descriptor for the detected feature points,

[0095] The improved SURF descriptor adds local direction consistency constraints on the basis of the traditional 64-dimensional descriptor. Then, based on the improved SURF descriptor, a hierarchical matching strategy is adopted to establish the correspondence between the feature points of the three images. The hierarchical matching strategy includes two stages: coarse matching and fine matching. The similarity metric in the coarse matching stage is for:

[0096]

[0097] in, and is the descriptor of the feature point to be matched, is the spatial distance of the feature points; the geometric constraints of the three views are introduced in the precise matching stage to define the matching confidence:

[0098]

[0099] in, is the three-view geometric error, is the scale factor; finally, abnormal matching points are removed based on the matching confidence to obtain a stable set of feature point correspondences.

[0100] S140: Calculating a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtaining depth information of the target scene using a triangulation method;

[0101] In one embodiment of the present invention, calculating the fundamental matrix and the intrinsic matrix of the image data according to the matching relationship specifically includes the following steps:

[0102] Firstly, a coordinate mapping relationship is established for the matching feature point pairs between the left camera image, the middle camera image and the right camera image, and each pair of matching points contains its two-dimensional image coordinate information and local image feature information; secondly, an improved basic matrix calculation method is adopted to solve the optimization objective function by introducing an adaptive weight, and the optimization objective function is defined as:

[0103]

[0104] in, is the basic matrix to be solved, and For the For the homogeneous coordinates of the matching points, is the number of valid matching point pairs, is the adaptive weight coefficient; then, based on the known camera calibration parameters, the basic matrix is ​​converted into an intrinsic matrix, which contains the relative rotation and translation information between the cameras. After obtaining the intrinsic matrix, the depth information of the target scene is calculated using the three-view depth fusion method.

[0105] S150: Fusing the depth information with the corrected image data to generate three-dimensional point cloud data with depth annotations.

[0106] In one embodiment of the present invention, the entire process, from the generation of personalized dialysis plans, the collection and encrypted transmission of physiological parameters, to the generation of in-depth analysis and nursing management reports, has achieved a high degree of intelligence and automation. This not only reduces the workload of medical staff, but also improves the accuracy and efficiency of the entire treatment process.

[0107] In the technical solution of the present invention, the target scene image is collected and distortion correction is performed through a multi-view camera group, and the depth information of the target scene is accurately calculated in combination with the improved SURF feature point detection and matching method. The depth estimation accuracy in different depth ranges is optimized through an asymmetric camera layout. The combination of double-layer distortion correction, feature point matching and triangulation not only improves the quality of image data, but also enhances the accuracy of depth information, and finally generates three-dimensional point cloud data with depth annotations, which has higher depth estimation accuracy and robustness and is suitable for scenes with a variety of depth ranges.

[0108] The following describes an embodiment of the device of the present invention, which can be used to execute the real-time data processing method of the three-eye camera for multi-view image acquisition in the above-mentioned embodiment of the present invention. It can be understood that the device can be a computer program (including program code) running in a computer device, for example, the device is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiment of the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the real-time data processing method of the three-eye camera for multi-view image acquisition in the above-mentioned embodiment of the present invention.

[0109] Figure 3 A block diagram of a three-eye camera real-time data processing system for multi-view image acquisition according to an embodiment of the present invention is shown.

[0110] Reference Figure 3 As shown, a three-eye camera real-time data processing system for multi-view image acquisition according to an embodiment of the present invention includes:

[0111] The image acquisition unit collects image data of the target scene through the left camera, the middle camera and the right camera.

[0112] A correction unit, performing a first distortion correction and a second distortion correction on the image data collected by the left camera, the middle camera and the right camera;

[0113] A feature detection unit, performing feature point detection based on the corrected image data to establish a matching relationship between the image data;

[0114] A feature processing unit calculates a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtains depth information of the target scene by using a triangulation method;

[0115] A fusion unit is used to fuse the depth information with the corrected image data to generate three-dimensional point cloud data with depth annotations.

[0116] In the technical solution of the present invention, the target scene image is collected and distortion correction is performed through a multi-view camera group, and the depth information of the target scene is accurately calculated in combination with the improved SURF feature point detection and matching method. The depth estimation accuracy in different depth ranges is optimized through an asymmetric camera layout. The combination of double-layer distortion correction, feature point matching and triangulation not only improves the quality of image data, but also enhances the accuracy of depth information, and finally generates three-dimensional point cloud data with depth annotations, which has higher depth estimation accuracy and robustness and is suitable for scenes with a variety of depth ranges.

[0117] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown.

[0118] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present invention.

[0119] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory 402 or the program loaded from the storage part 408 to the random access memory 403, such as executing the three-eye camera real-time data processing method for multi-view image acquisition described in the above embodiment. In the random access memory 403, various programs and data required for system operation are also stored. The central processing unit 401, the read-only memory 402 and the random access memory 403 are connected to each other via a bus 404. The input / output interface 405 is also connected to the bus 404.

[0120] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0121] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the system of the present invention are executed.

[0122] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0125] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0126] As another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the three-eye camera real-time data processing method for multi-view image acquisition described in the above embodiment.

[0127] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0128] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0129] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the embodiments disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention.

[0130] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A three-eye camera real-time data processing method for multi-view image acquisition, characterized in that: include: The image data of the target scene is collected by the left camera, the middle camera and the right camera. Performing a first distortion correction and a second distortion correction on the image data collected by the left camera, the middle camera and the right camera; Perform feature point detection based on the corrected image data and establish a matching relationship between the image data; Calculating a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtaining depth information of the target scene by using a triangulation method; Fusion of the depth information with the corrected image data to generate three-dimensional point cloud data with depth annotations; The feature point detection includes constructing a scale space pyramid for the corrected left camera image, the middle camera image and the right camera image, respectively, and performing feature point detection at each scale space layer; The feature point detection uses the Hessian matrix, as shown in the following formula: in, , , is the second-order derivative of the corrected image in the corresponding direction, is the adaptive weight matrix based on image content: in, is the image gradient amplitude, and To adjust the parameters; The matching relationship between the image data is established by adopting a hierarchical matching strategy; the hierarchical matching strategy includes coarse matching and fine matching; The similarity metric of the rough matching stage The calculation of is as follows: in, and is the descriptor of the feature point to be matched, is the spatial distance of feature points, is the Gaussian kernel parameter of the distance weight, and represents the feature points to be matched; The matching confidence of the fine matching stage The calculation of is as follows: in, is the three-view geometric error, is the scale factor, is the coarse matching similarity between points p and q, is the coarse matching similarity between points q and r.

2. The three-eye camera real-time data processing method for multi-view image acquisition according to claim 1, characterized in that: The left camera and the middle camera are fixedly installed on the same horizontal plane and maintain a first preset distance; the middle camera and the right camera are fixedly installed on the same horizontal plane and maintain a second preset distance; The optical axes of the left camera, the middle camera and the right camera are arranged in parallel and perpendicular to the same horizontal plane, and the left camera, the middle camera and the right camera synchronously collect image data of the target scene through a synchronous trigger signal.

3. The real-time data processing method of a three-eye camera for multi-view image acquisition according to claim 1, characterized in that: The first distortion correction comprises: Determine the optical center position of the image, and then calculate the distance from each pixel to the optical center; based on the distance from each pixel to the optical center, use a polynomial model to model the distortion; at the same time, different radial distortion coefficients control the power terms of the polynomial model, and work together to correct the distortion effect; The second distortion correction comprises: The result of the first distortion correction is further optimized to correct the degree of lens tilt at different temperatures.

4. The three-eye camera real-time data processing method for multi-view image acquisition according to claim 3 is characterized in that: The first distortion correction also includes adaptive weights and Gaussian attenuation terms; The adaptive weight dynamically adjusts the correction strength by calculating the depth estimation error of the feature point; The Gaussian attenuation term allows the correction effect to transition smoothly as the distance increases, thereby avoiding sudden changes or unnaturalness in the edge areas of the image during the correction process.

5. The real-time data processing method of a three-eye camera for multi-view image acquisition according to claim 4, characterized in that: Constructing a radial distortion correction matrix and a tangential distortion correction matrix based on the results of the first distortion correction and the second distortion correction; Applying the radial distortion correction matrix and the tangential distortion correction matrix in cascade to the original image data to obtain corrected image data; The radial distortion correction matrix is ​​shown below: in, and are the focal length parameters of the camera in the x and y directions, is the optical center coordinate, is the square of the distance from the pixel to the optical center, is the radial distortion coefficient, is the adaptive weight coefficient, is the Gaussian attenuation factor; The tangential distortion correction matrix is ​​shown below: in, is the first tangential distortion coefficient, Describes the cross distortion in the xy plane, is the second tangential distortion coefficient, describes the cross distortion in the other direction, is the original image coordinate, is the tilt coefficient after temperature compensation, is the current temperature, is the temperature compensation function.

6. A three-eye camera real-time data processing system for multi-view image acquisition, characterized in that: include: The image acquisition unit collects image data of the target scene through the left camera, the middle camera and the right camera. A correction unit, performing a first distortion correction and a second distortion correction on the image data collected by the left camera, the middle camera and the right camera; A feature detection unit performs feature point detection based on the corrected image data and establishes a matching relationship between the image data; A feature processing unit calculates a fundamental matrix and an intrinsic matrix of the image data according to the matching relationship, and obtains depth information of the target scene by using a triangulation method; A fusion unit, which fuses the depth information with the corrected image data to generate three-dimensional point cloud data with depth annotation; The feature point detection uses the Hessian matrix, as shown in the following formula: in, , , is the second-order derivative of the corrected image in the corresponding direction, is the adaptive weight matrix based on image content: in, is the image gradient amplitude, and To adjust the parameters; The matching relationship between the image data is established by adopting a hierarchical matching strategy; the hierarchical matching strategy includes coarse matching and fine matching; The similarity metric of the rough matching stage The calculation of is as follows: in, and is the descriptor of the feature point to be matched, is the spatial distance of feature points, is the Gaussian kernel parameter of the distance weight, and represents the feature points to be matched; The matching confidence of the fine matching stage The calculation of is as follows: in, is the three-view geometric error, is the scale factor, is the coarse matching similarity between points p and q, is the coarse matching similarity between points q and r.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the real-time data processing method of a three-eye camera for multi-view image acquisition according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the real-time data processing method of a three-eye camera for multi-view image acquisition as described in any one of claims 1 to 5.

Images