In-vehicle scene construction method, terminal device and computer readable storage medium

Abstract

The application is suitable for the field of visual technology, and provides a vehicle interior scene construction method and device, terminal equipment and computer readable storage medium, which comprises the following steps: acquiring a first image of a vehicle interior space by a first shooting device at a first shooting position; acquiring a second image of the vehicle interior space by a second shooting device at a second shooting position; calculating a first transformation matrix between a first camera coordinate system to which the first shooting device belongs and a second camera coordinate system to which the second shooting device belongs according to first feature points and second feature points, wherein the first feature points are feature points in the first image, and the second feature points are feature points corresponding to the first feature points in the second image; calculating three-dimensional coordinates of a first space point of the vehicle interior space according to the first transformation matrix; and constructing a three-dimensional scene of the vehicle interior space according to the three-dimensional coordinates of the first space point. Through the above method, an accurate three-dimensional scene in the vehicle interior can be constructed, and the position of a target object in the cabin can be accurately determined.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an in-vehicle scene construction method and device, a terminal device, and a computer readable storage medium. BACKGROUND

[0002] Visual technology is widely used in the field of automobiles. For example, the in-vehicle environment and the in-vehicle personnel state are monitored through the in-vehicle camera. However, in the prior art, the image collected by the in-vehicle camera is usually a two-dimensional image, which lacks depth information. The target object in the cabin is in a three-dimensional space, and therefore, the existing method cannot determine the accurate position of the target object in the cabin. SUMMARY

[0003] The embodiments of the present application provide an in-vehicle scene construction method, device, terminal device, and computer readable storage medium, which can construct an accurate in-vehicle three-dimensional scene, and thus facilitate accurate determination of the position of the target object in the cabin.

[0004] In a first aspect, the embodiments of the present application provide an in-vehicle scene construction method, comprising:

[0005] obtaining a first image of an in-vehicle space through a first shooting device at a first shooting position;

[0006] obtaining a second image of the in-vehicle space through a second shooting device at a second shooting position;

[0007] calculating a first transformation matrix between a first camera coordinate system to which the first shooting device belongs and a second camera coordinate system to which the second shooting device belongs according to a first feature point and a second feature point, wherein the first feature point is a feature point in the first image, the second feature point is a feature point in the second image corresponding to the first feature point, and the first transformation matrix is a transformation matrix between the first camera coordinate system and the second camera coordinate system;

[0008] calculating a three-dimensional coordinate of a first space point of the in-vehicle space according to the first transformation matrix, wherein the first space point is a space point corresponding to the first feature point and the second feature point;

[0009] constructing a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point.

[0010] In a possible implementation manner of the first aspect, the calculating the first transformation matrix between the first camera coordinate system to which the first shooting device belongs and the second camera coordinate system to which the second shooting device belongs according to the first feature point and the second feature point comprises:

[0011] performing feature point matching processing on the first image and the second image to obtain the first feature point and the second feature point.

[0012] convert the first feature point into a first coordinate point and convert the second feature point into a second coordinate point according to a preset intrinsic matrix, wherein the intrinsic matrix represents a conversion relationship between an image coordinate system and a camera coordinate system;

[0013] calculate the first transformation matrix according to the first coordinate point and the second coordinate point.

[0014] In a possible implementation manner of the first aspect, the calculating the first transformation matrix according to the first coordinate point and the second coordinate point comprises:

[0015] calculating an essential matrix according to the first coordinate point and the second coordinate point, wherein the essential matrix represents a conversion relationship between the first camera coordinate system and the second camera coordinate system;

[0016] decomposing the first transformation matrix from the essential matrix.

[0017] In a possible implementation manner of the first aspect, the calculating the three-dimensional coordinate of the first space point of the in-vehicle space according to the first transformation matrix comprises:

[0018] obtaining a first distance between a position of the first space point in the in-vehicle space and the second shooting device;

[0019] calculating the three-dimensional coordinate of the first space point according to the first distance, the intrinsic matrix and the first transformation matrix.

[0020] In a possible implementation manner of the first aspect, the constructing the three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point comprises:

[0021] obtaining a third image of the in-vehicle space, wherein the third image is obtained by a third shooting device;

[0022] determining a second transformation matrix according to a third feature point and a fourth feature point, wherein the third feature point is a feature point in the second image, the fourth feature point is a feature point corresponding to the third feature point in the third image, and the second transformation matrix is a transformation matrix between a third camera coordinate system to which the third shooting device belongs and a first camera coordinate system to which the first shooting device belongs;

[0023] calculating a three-dimensional coordinate of a second space point of the in-vehicle space according to the second transformation matrix, wherein the second space point is a space point corresponding to the third feature point and the fourth feature point;

[0024] construct a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinates of the first space point and the three-dimensional coordinates of the second space point.

[0025] In a possible implementation of the first aspect, the third feature point and the fourth feature point are used to determine a second transformation matrix between a third camera coordinate system to which the third photographing device belongs and a first camera coordinate system to which the first photographing device belongs, and the second transformation matrix comprises:

[0026] The third photographing position of the third photographing device is calculated according to the three-dimensional coordinates of a space point corresponding to a fifth feature point, where the fifth feature point is a feature point belonging to the second feature point in the third feature point.

[0027] The second transformation matrix between the third camera coordinate system corresponding to the third photographing position and the first camera coordinate system corresponding to the first photographing position is calculated.

[0028] In a possible implementation of the first aspect, after the three-dimensional scene of the in-vehicle space is constructed according to the three-dimensional coordinates of the first space point and the three-dimensional coordinates of the second space point, the method comprises:

[0029] A loss value corresponding to a scene construction parameter is calculated according to a preset cost function, where the scene construction parameter comprises the intrinsic parameter matrix, the transformation matrix corresponding to each photographing position, the three-dimensional coordinates of a space point in the in-vehicle space, and pixel coordinates of the space point in each image in the in-vehicle space.

[0030] The scene construction parameter is optimized according to a gradient descent method and the loss value.

[0031] In a second aspect, an in-vehicle scene construction device is provided, comprising:

[0032] A first acquisition unit is configured to acquire a first image of an in-vehicle space by using a first photographing device at a first photographing position.

[0033] A second acquisition unit is configured to acquire a second image of the in-vehicle space by using a second photographing device at a second photographing position.

[0034] A first calculation unit is configured to calculate a first transformation matrix according to a first feature point and a second feature point, where the first feature point is a feature point in the first image, the second feature point is a feature point corresponding to the first feature point in the second image, and the first transformation matrix is a transformation matrix between a first camera coordinate system to which the first photographing device belongs and a second camera coordinate system to which the second photographing device belongs.

[0035] a second computing unit, configured to calculate three-dimensional coordinates of a first space point of the in-vehicle space according to the first transformation matrix, wherein the first space point is a space point corresponding to the first feature point and the second feature point;

[0036] a scene constructing unit, configured to construct a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinates of the first space point.

[0037] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and characterized in that the processor implements the in-vehicle scene constructing method according to any one of the first aspect when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and characterized in that the computer program is executed by a processor to implement the in-vehicle scene constructing method according to any one of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, causes the terminal device to execute the in-vehicle scene constructing method according to any one of the first aspect.

[0040] In the embodiment of the present application, the in-vehicle space is photographed by the photographing devices at different positions to obtain the first image and the second image; the transformation matrix between the camera coordinate systems to which the different photographing devices respectively belong is calculated according to the first image and the second image obtained at different photographing positions; the three-dimensional coordinates of the space points of the in-vehicle space are calculated according to the transformation matrix; and finally, the three-dimensional scene of the in-vehicle space is constructed according to the three-dimensional coordinates of the space points of the in-vehicle space. Through the above method, an accurate in-vehicle three-dimensional scene can be constructed under the condition of lacking depth information, which is conducive to accurately determining the position of the target object in the cabin.

[0041] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1is a flowchart of a method for constructing an in-vehicle scene provided by an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of photographing of two positions provided by an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a calibration pattern provided by an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a camera position in a vehicle cabin provided by an embodiment of the present application;

[0047] Figure 5 is a structural block diagram of an in-vehicle scene construction apparatus provided by an embodiment of the present application;

[0048] Figure 6 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc., in order to provide a thorough understanding of the present embodiments. However, it will be apparent to those skilled in the art that the present embodiments can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, and circuits are omitted so as not to obscure the description of the present embodiments.

[0050] It is to be understood that the terminology “includes,” “has,” “holds,” “contains,” “comprises,” “comprised of,” “comprising,” or variants thereof does not exclude the presence of additional

[0051] It is to be understood that the terminology “includes,” “has,” “holds,” “contains,” “comprises,” “comprised of,” “comprising,” or variants thereof does not exclude the presence of additional

[0052] As used in the present specification and claims, the term “if’ can be interpreted as meaning “when” or “once” or “in response to a determination” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted as meaning “once it is determined” or “in response to the determination” or “once [a described condition or event] is detected” or “in response to the detection of [a described condition or event]” depending on the context.

[0053] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0054] In the present application, the reference to "one embodiment" or "some embodiments" means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0055] Referring to Figure 1 is a flowchart of the in-vehicle scene construction method provided by the embodiments of the present application, which is an example and not a limitation. The method can include the following steps:

[0056] S101, acquiring a first image of the in-vehicle space by a first shooting device at a first shooting position.

[0057] S102, acquiring a second image of the in-vehicle space by a second shooting device at a second shooting position.

[0058] In the embodiments of the present application, the first shooting device and the second shooting device can refer to different shooting devices installed at different shooting positions, or refer to the same shooting device placed at different shooting positions.

[0059] Referring to Figure 2 is a schematic diagram of shooting at two positions provided by the embodiments of the present application. As shown in Figure 2 , p represents a space point in the in-vehicle space, o1xyz is a first camera coordinate system to which the first shooting position belongs, and o2xyz is a second camera coordinate system to which the second shooting position belongs. x1 is a pixel point (i.e. a first feature point) corresponding to p in the first image, and x2 is a pixel point (i.e. a second feature point) corresponding to p in the second image. It should be noted that Figure 2 only one space point is shown in , and only one feature point is marked in the corresponding first image and second image. In fact, each of the first image and the second image includes a plurality of feature points, each of which corresponds to a space point in the in-vehicle space.

[0060] S103, calculating a first transformation matrix according to the first feature point and the second feature point.

[0061] The first feature point is a feature point in the first image, the second feature point is a feature point corresponding to the first feature point in the second image, and the first transformation matrix is a transformation matrix between a first camera coordinate system to which the first photographing device belongs and a second camera coordinate system to which the second photographing device belongs.

[0062] It should be noted that, since the first transformation matrix includes a plurality of parameters, a plurality of groups of feature points need to be used for calculation. Each group of feature points includes a first feature point and a second feature point corresponding to the first feature point. The calculation process of any group of feature points is described below.

[0063] In an embodiment, the calculation manner of the first transformation matrix includes:

[0064] S1031, performing feature point matching processing on the first image and the second image to obtain the first feature point and the second feature point.

[0065] Optionally, the existing Scale-invariant feature transform (SIFT) algorithm or Speeded Up Robust Features (SUFT) algorithm can be used for the feature point matching processing.

[0066] S1032, converting the first feature point into a first coordinate point and converting the second feature point into a second coordinate point according to a preset intrinsic matrix, wherein the intrinsic matrix represents a conversion relationship between an image coordinate system and a camera coordinate system.

[0067] The intrinsic matrix in the embodiment of the present application is pre-calibrated. The specific process is as follows: a photographing device is used to photograph a preset calibration pattern, and the intrinsic matrix is calculated according to the coordinates of the pixel points in the photographed image. For details, refer to Figure 3 is a schematic diagram of the calibration pattern provided by the embodiment of the present application. The intrinsic matrix K is:

[0068]

[0069] wherein f is a camera focal length, c x is a horizontal coordinate of the center of the image, and c y is a vertical coordinate of the center of the image.

[0070] After the intrinsic matrix is obtained, the first coordinate point and the second coordinate point are calculated according to the following formula:

[0071] K -1 x1=x′1;

[0072] K -1 x2=x′2;

[0073] wherein x1 and x2 are corresponding points on the first image and the second image respectively, and x'1 and x'2 are the transformed first coordinate point and the second coordinate point respectively. It should be noted that when the first photographing device and the second photographing device are the same device, K in the above two formulas is the same; when the first photographing device and the second photographing device are different devices, K in the above two formulas respectively represents the intrinsic matrix corresponding to the first photographing device and the intrinsic matrix corresponding to the second photographing device.

[0074] S1033, calculating the first transformation matrix according to the first coordinate point and the second coordinate point.

[0075] In the above manner, since the intrinsic matrix is calibrated in advance, the calculation time of the first transformation matrix can be effectively saved.

[0076] Optionally, one implementation of S1033 comprises:

[0077] calculating an essential matrix according to the first coordinate point and the second coordinate point; and decomposing the first transformation matrix from the essential matrix. Wherein the essential matrix represents the conversion relationship between the first camera coordinate system and the second camera coordinate system.

[0078] For example, it is assumed that the first camera coordinate system is determined as the world coordinate system. According to the epipolar constraint equation

[0079] x'1Ex'2=0

[0080] calculating the essential matrix E. Since the essential matrix E contains 5 parameters of rotation and translation, at least 5 pairs of first coordinate points and second coordinate points are required to be substituted into the above epipolar constraint equation to calculate the essential matrix. After the essential matrix is calculated, the essential matrix can be decomposed (such as SVD decomposition) to obtain the first transformation matrix. Wherein the first transformation matrix includes a rotation matrix R2 and a translation matrix T2.

[0081] In the above manner, the first transformation matrix can be obtained by using a simple mathematical method (i.e. matrix decomposition). Since the calculation method of matrix decomposition is relatively simple, the calculation complexity can be effectively reduced.

[0082] S104, calculating the three-dimensional coordinates of the first space point of the in-vehicle space according to the first transformation matrix.

[0083] wherein the first space point is the space point corresponding to the first feature point and the second feature point.

[0084] In one embodiment, S104 comprises:

[0085] obtaining a first distance between the position of the first spatial point in the vehicle interior space and the second photographing device; and calculating the three-dimensional coordinate of the first spatial point according to the first distance, the intrinsic parameter matrix and the first transformation matrix.

[0086] In the embodiments of the present application, the first distance can be obtained by laser ranging assistance.

[0087] For example, a model equation of the second photographing device is established as follows:

[0088] s2x2=K(R2X+T2);

[0089] wherein s2 is the first distance, x2 is a second feature point in the second image, K is an intrinsic parameter matrix, R2 and T2 are the first transformation matrix, and X is the three-dimensional coordinate of the first spatial point corresponding to the second feature point x2. The above formula can be transformed as follows:

[0090]

[0091] The zero space of the matrix can be obtained by using the SVD algorithm. Through the above method, the three-dimensional coordinates of the first spatial points corresponding to each second feature point in the second image are calculated respectively.

[0092] Through the above method, the three-dimensional coordinates can be calculated by using a simple mathematical method (i.e., matrix calculation). Since the matrix calculation is relatively simple, the calculation complexity can be effectively reduced.

[0093] S105, constructing a three-dimensional scene of the vehicle interior space according to the three-dimensional coordinates of the first spatial points.

[0094] According to the three-dimensional coordinates of the plurality of first spatial points, the positions of the plurality of first spatial points in the vehicle interior space can be determined, and then the three-dimensional scene of the vehicle interior space can be constructed by the plurality of first spatial points.

[0095] In the embodiments of the present application, the vehicle interior space is photographed by photographing devices at different positions to obtain the first image and the second image; the transformation matrix between the camera coordinate systems to which the different photographing devices belong is calculated according to the first image and the second image obtained at different photographing positions; the three-dimensional coordinates of the spatial points of the vehicle interior space are calculated according to the transformation matrix; and finally, the three-dimensional scene of the vehicle interior space is constructed according to the three-dimensional coordinates of the spatial points of the vehicle interior space. Through the above method, an accurate three-dimensional scene of the vehicle interior can be constructed under the condition of lacking depth information, which is beneficial to accurately determining the position of the target object in the cabin.

[0096] In order to improve the accuracy of the three-dimensional scene construction, in one embodiment, a multi-position joint calculation method is proposed. Specifically, S105 includes:

[0097] S1051, acquire a third image of the in-vehicle space, the third image being acquired by a third photographing device.

[0098] In the embodiments of the present application, the third photographing device and the first photographing device and the second photographing device can refer to different photographing devices installed at different photographing positions, or can refer to the same photographing device placed at different photographing positions.

[0099] S1052, determine a second transformation matrix according to the third feature point and a fourth feature point.

[0100] The third feature point is a feature point in the second image, the fourth feature point is a feature point in the third image corresponding to the third feature point, and the second transformation matrix is between a third camera coordinate system to which the third photographing device belongs and a first camera coordinate system to which the first photographing device belongs. The same as step S1031, the existing feature matching algorithm can be used to perform feature point matching processing on the third image and the second image to obtain the third feature point and the fourth feature point. Similarly, the second image includes a plurality of third feature points, and correspondingly, the third image includes a plurality of fourth feature points matched with the plurality of third feature points respectively.

[0101] It should be noted that the third feature point obtained after the feature point matching processing of the second image and the third image can include a fifth feature point, and the fifth feature point is a feature point in the third feature point that belongs to the second feature point. In other words, part of the feature points in the second image are matched with the feature points in the first image and the feature points in the third image, and this part of the feature points is the fifth feature point. Another part of the feature points in the second image are not matched with the feature points in the first image but are matched with the feature points in the third image, and this part of the feature points is a newly added feature point. Since the three-dimensional coordinates corresponding to the fifth feature point are determined by the above steps, the position of the third photographing device can be corrected according to the fifth feature point.

[0102] Optionally, the calculation method of the second transformation matrix is as follows:

[0103] According to the three-dimensional coordinates of the space point corresponding to the fifth feature point, the third photographing position of the third photographing device is calculated, and the second transformation matrix between the third camera coordinate system corresponding to the third photographing position and the first camera coordinate system corresponding to the first photographing position is calculated.

[0104] In actual application, the calculation of the above-mentioned second transformation matrix can be realized by using the solvePnP tool package in OpenCV.

[0105] S1053, calculating three-dimensional coordinates of a second space point of the in-vehicle space according to the second transformation matrix, wherein the second space point is a space point corresponding to the third feature point and the fourth feature point.

[0106] As described in S1052, the fifth feature point and the new feature point are included in the second image. After the position of the third shooting device is corrected according to the method of S1052, the three-dimensional coordinates of the space point corresponding to the fifth feature point in the second space point are consistent with the three-dimensional coordinates of the space point corresponding to the fifth feature point in the first space point. The space points in the second space point other than the space point corresponding to the fifth feature point are space points corresponding to the new feature points, which means that more three-dimensional coordinates of space points are determined through the third image.

[0107] S1054, constructing a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinates of the first space point and the three-dimensional coordinates of the second space point of the in-vehicle space.

[0108] Of course, after S1054, images of the in-vehicle space corresponding to multiple shooting positions can be continuously introduced, and three-dimensional coordinates of multiple space points of the in-vehicle space can be calculated jointly from the images of the in-vehicle space corresponding to multiple shooting positions. The calculation method of the next shooting position is the same as the method of S1051-S1053, which will not be described here.

[0109] In the embodiments of the present application, by introducing multiple shooting positions and jointly calculating multiple shooting positions, the cumulative error of the position can be effectively reduced, and the accuracy of the three-dimensional scene construction can be effectively improved.

[0110] It should be noted that the more shooting positions are introduced, the more space points can be determined, the more accurate three-dimensional coordinates of the space points can be calculated, and the more accurate three-dimensional scene can be created.

[0111] Referring to Figure 4 is a schematic diagram of camera positions in a vehicle cabin provided by an embodiment of the present application. As shown in Figure 4 , eight shooting positions are set in the vehicle cabin. In actual applications, the number and specific positions of the shooting positions are not limited.

[0112] With the increase of the introduced shooting positions, the accumulated error will also increase. In order to reduce the accumulated error, in an embodiment, after S1054, the method further comprises:

[0113] According to a preset cost function, a loss value corresponding to a scene construction parameter is calculated, wherein the scene construction parameter comprises the intrinsic matrix, a transformation matrix corresponding to each shooting position, three-dimensional coordinates of a space point in the vehicle space, and pixel coordinates of the space point in each image in the vehicle space; and the scene construction parameter is optimized according to a gradient descent method and the loss value.

[0114] Wherein, the cost function can be f i (K i ,R i ,T i ,P j )=π(K i [R i T i ]P i ]-p j i ), K i is an intrinsic matrix of the i th shooting device, R i and T i are a transformation matrix corresponding to the i th shooting position, P j is a coordinate of a certain space point in the vehicle space, and p j is a pixel coordinate of the point in the i th image. i , π is a projection function. It should be noted that when one shooting device is used, the shooting device is moved to different shooting positions to obtain images of the vehicle space, K i and K i+1 are the same intrinsic matrix in the above formula.

[0115] According to min x ∑ i ρ i (||f i (x i1 ,x i2 ,…,x ik )||) 2 is optimized. Wherein, ρ i is a loss function corresponding to the i th shooting position.

[0116] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0117] Corresponding to the vehicle interior scene construction method described in the above embodiment, Figure 5 is a structural block diagram of a vehicle interior scene construction device provided by the embodiments of the application, and only the part related to the embodiments of the application is shown for ease of description.

[0118] Referring toFigure 5 The device comprises:

[0119] A first acquisition unit 51 is configured to acquire a first image of an in-vehicle space by a first photographing device at a first photographing position.

[0120] A second acquisition unit 52 is configured to acquire a second image of the in-vehicle space by a second photographing device at a second photographing position.

[0121] A first calculation unit 53 is configured to calculate a first transformation matrix according to a first feature point and a second feature point, wherein the first feature point is a feature point in the first image, the second feature point is a feature point in the second image corresponding to the first feature point, and the first transformation matrix is a transformation matrix between a first camera coordinate system to which the first photographing device belongs and a second camera coordinate system to which the second photographing device belongs.

[0122] A second calculation unit 54 is configured to calculate a three-dimensional coordinate of a first space point of the in-vehicle space according to the first transformation matrix, wherein the first space point is a space point corresponding to the first feature point and the second feature point.

[0123] A scene construction unit 55 is configured to construct a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point.

[0124] Optionally, the first calculation unit 53 is further configured to:

[0125] perform feature point matching processing on the first image and the second image to obtain the first feature point and the second feature point;

[0126] convert the first feature point into a first coordinate point and the second feature point into a second coordinate point according to a preset intrinsic matrix, wherein the intrinsic matrix represents a conversion relationship between an image coordinate system and a camera coordinate system;

[0127] calculate the first transformation matrix according to the first coordinate point and the second coordinate point.

[0128] Optionally, the first calculation unit 53 is further configured to:

[0129] calculate an essential matrix according to the first coordinate point and the second coordinate point, wherein the essential matrix represents a conversion relationship between the first camera coordinate system and the second camera coordinate system;

[0130] extract the first transformation matrix from the essential matrix.

[0131] Optionally, the second calculation unit 54 is further configured to:

[0132] acquire a first distance between the first spatial point and the second photographing device in the in-vehicle space;

[0133] calculate the three-dimensional coordinate of the first spatial point according to the first distance, the intrinsic matrix and the first transformation matrix.

[0134] Optionally, the scene construction unit 55 is further configured to:

[0135] acquire a third image of the in-vehicle space, the third image being obtained by a third photographing device;

[0136] determine a second transformation matrix according to a third feature point and a fourth feature point, wherein the third feature point is a feature point in the second image, the fourth feature point is a feature point in the third image corresponding to the third feature point, and the second transformation matrix is a transformation matrix between a third camera coordinate system to which the third photographing device belongs and a first camera coordinate system to which the first photographing device belongs;

[0137] calculate a three-dimensional coordinate of a second spatial point of the in-vehicle space according to the second transformation matrix, wherein the second spatial point is a spatial point corresponding to the third feature point and the fourth feature point;

[0138] construct a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first spatial point and the three-dimensional coordinate of the second spatial point of the in-vehicle space.

[0139] Optionally, the scene construction unit 55 is further configured to:

[0140] calculate a third photographing position of the third photographing device according to a three-dimensional coordinate of a spatial point corresponding to a fifth feature point, wherein the fifth feature point is a feature point in the third feature point that belongs to the second feature point;

[0141] calculate the second transformation matrix between the third camera coordinate system corresponding to the third photographing position and the first camera coordinate system corresponding to the first photographing position.

[0142] Optionally, the apparatus 5 further comprises:

[0143] The parameter optimization unit 56 is configured to, after constructing the three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first spatial point and the three-dimensional coordinate of the second spatial point of the in-vehicle space, calculate a loss value corresponding to a scene construction parameter according to a preset cost function, wherein the scene construction parameter comprises the intrinsic matrix, a transformation matrix corresponding to each photographing position, a three-dimensional coordinate of a spatial point in the in-vehicle space, and a pixel coordinate of a spatial point in the in-vehicle space in each image; and optimize the scene construction parameter according to a gradient descent method and the loss value.

[0144] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0145] In addition, Figure 5 The in-vehicle scene construction apparatus shown can be a software unit, a hardware unit or a software and hardware combined unit built into an existing terminal device, can be integrated into the terminal device as an independent plug-in, or can exist as an independent terminal device.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above described functions. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0147] Figure 6 is a structural schematic diagram of a terminal device provided by the present application. As Figure 6 shown, the terminal device 6 of this embodiment includes at least one processor 60 Figure 6 only one processor is shown), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 executes the computer program 62 to implement the steps in any of the above in-vehicle scene construction method embodiments.

[0148] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The terminal device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 6 This is only an example of the terminal device 6 and does not constitute a limitation on the terminal device 6, which can include more or fewer components than shown, or combine certain components, or different components, for example, it can also include input / output devices, network access devices, etc.

[0149] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0150] The memory 61 can be an internal storage unit of the terminal device 6 in some embodiments, for example, a hard disk or a memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 61 can include both an internal storage unit and an external storage device of the terminal device 6. The memory 61 is used to store an operating system, an application program, a boot loader, data, and other programs, for example, program codes of the computer program, etc. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0151] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0152] The embodiments of the present application provide a computer program product. When the computer program product is run on a terminal device, the terminal device is caused to implement the steps in the above-mentioned various method embodiments.

[0153] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods through a computer program to instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0154] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0155] Those of ordinary skill in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0156] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the above-described apparatus / terminal equipment embodiments are only schematic. The division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between each displayed or discussed unit can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0157] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0158] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An in-vehicle scene construction method characterized by comprising: The method comprises: acquiring a first image of the in-vehicle space through a first photographing device at a first photographing position; acquiring a second image of the in-vehicle space through a second photographing device at a second photographing position; calculating a first transformation matrix according to a first feature point and a second feature point, wherein the first feature point is a feature point in the first image, the second feature point is a feature point in the second image corresponding to the first feature point, and the first transformation matrix is a transformation matrix between a first camera coordinate system to which the first photographing device belongs and a second camera coordinate system to which the second photographing device belongs; calculating a three-dimensional coordinate of a first space point of the in-vehicle space according to the first transformation matrix, wherein the first space point is a space point corresponding to the first feature point and the second feature point; constructing a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point. In the method, one photographing device is used, and the photographing device is moved to different photographing positions to acquire images of the in-vehicle space. The method further comprises the following steps for improving the construction accuracy of the three-dimensional scene: calculating a loss value corresponding to a scene construction parameter according to a preset cost function, wherein the scene construction parameter comprises an intrinsic matrix, a transformation matrix corresponding to each photographing position, a three-dimensional coordinate of a space point in the in-vehicle space, and a pixel coordinate of the space point in each image; optimizing the scene construction parameter according to a gradient descent method and the loss value.

2. The in-vehicle scene building method according to claim 1, wherein The method of calculating the first transformation matrix between the first camera coordinate system to which the first photographing device belongs and the second camera coordinate system to which the second photographing device belongs according to the first feature point and the second feature point comprises: performing feature point matching processing on the first image and the second image to obtain the first feature point and the second feature point; converting the first feature point into a first coordinate point and converting the second feature point into a second coordinate point according to a preset intrinsic matrix, wherein the intrinsic matrix represents a conversion relationship between an image coordinate system and a camera coordinate system; calculating the first transformation matrix according to the first coordinate point and the second coordinate point.

3. The in-vehicle scene construction method according to claim 2, characterized by, The method of calculating the first transformation matrix according to the first coordinate point and the second coordinate point comprises: calculating an essential matrix according to the first coordinate point and the second coordinate point, wherein the essential matrix represents a conversion relationship between the first camera coordinate system and the second camera coordinate system; decomposing the first transformation matrix from the essential matrix.

4. The in-vehicle scene construction method according to claim 2, characterized by, The method of calculating the three-dimensional coordinate of the first space point of the in-vehicle space according to the first transformation matrix comprises: acquiring a first distance between a position of the first space point in the in-vehicle space and the second photographing device; calculating the three-dimensional coordinate of the first space point according to the first distance, the intrinsic matrix, and the first transformation matrix.

5. The in-vehicle scene construction method according to claim 1, wherein The method of constructing the three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point comprises: acquiring a third image of the in-vehicle space, wherein the third image is acquired by a third photographing device; determining a second transformation matrix according to a third feature point and a fourth feature point, wherein the third feature point is a feature point in the second image, the fourth feature point is a feature point in the third image corresponding to the third feature point, and the second transformation matrix is a transformation matrix between a third camera coordinate system to which the third photographing device belongs and a first camera coordinate system to which the first photographing device belongs; calculating a three-dimensional coordinate of a second space point of the in-vehicle space according to the second transformation matrix, wherein the second space point is a space point corresponding to the third feature point and the fourth feature point; constructing a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point and the three-dimensional coordinate of the second space point.

6. The in-vehicle scene construction method according to claim 5, characterized by, The method further includes: calculating a third photographing position of the third photographing device according to a three-dimensional coordinate of a space point corresponding to a fifth feature point, wherein the fifth feature point is a feature point in the third feature point that belongs to the second feature point; and calculating the second transformation matrix between the third camera coordinate system corresponding to the third photographing position and the first camera coordinate system corresponding to the first photographing position.

7. An in-vehicle scene construction apparatus characterized by comprising: The method further includes: a first obtaining unit configured to obtain a first image of an in-vehicle space by a first photographing device at a first photographing position; a second obtaining unit configured to obtain a second image of the in-vehicle space by a second photographing device at a second photographing position; a first calculating unit configured to calculate a first transformation matrix according to a first feature point and a second feature point, wherein the first feature point is a feature point in the first image, the second feature point is a feature point in the second image corresponding to the first feature point, and the first transformation matrix is a transformation matrix between a first camera coordinate system to which the first photographing device belongs and a second camera coordinate system to which the second photographing device belongs; a second calculating unit configured to calculate a three-dimensional coordinate of a first space point of the in-vehicle space according to the first transformation matrix, wherein the first space point is a space point corresponding to the first feature point and the second feature point; a scene constructing unit configured to construct a three-dimensional scene of the in-vehicle space according to the three-dimensional coordinate of the first space point. The method further includes: moving one photographing device to different photographing positions to obtain images of the in-vehicle space; The method further includes steps to improve the construction accuracy of the three-dimensional scene, including: calculating a loss value corresponding to a scene construction parameter according to a preset cost function, wherein the scene construction parameter includes an intrinsic matrix, a transformation matrix corresponding to each photographing position, a three-dimensional coordinate of a space point in the in-vehicle space, and a pixel coordinate of the space point in each image; 8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, optimizing the scene construction parameter according to a gradient descent method and the loss value. The processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, which is executed by a processor, implements the method as claimed in any of claims 1 to 6.

Images