Image processing method, vehicle-mounted equipment and storage medium
By generating the transformation matrix to splice vehicle images from different perspectives, the problem of inaccurate detection caused by the lack of vehicle information is solved, and the vehicle tracking effect is improved.
Patent Information
- Application Number
- CN202311873203.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2025-07-01
AI Technical Summary
Due to the limitations of the position and angle of the vehicle shooting equipment, the captured external vehicle images only contain part of the vehicle body. Direct detection may lead to the loss of vehicle information and affect the vehicle tracking effect.
By acquiring the upper body images and lower body images of different shooting devices, multiple vertex spatial coordinates of the target viewing plane are determined, and projected into their respective image coordinate systems, a transformation matrix is generated to splice the complete body images, and vehicle information detection is performed using the target detection model.
Ensure the accuracy of vehicle detection results and vehicle tracking effect, and generate a complete body image to improve detection accuracy.
Smart Images
Figure CN120235987A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving technology, and particularly to an image processing method, an in-vehicle device, and a storage medium. Background Art
[0002] During vehicle driving, for the sake of safety and intelligent driving, external vehicle images around the vehicle are often obtained through a photographing device installed on the vehicle. However, due to limitations such as the photographing position and the photographing angle, the captured images may only include partial body parts of the external vehicle. In related technologies, the captured partial body images are often directly detected, and the external vehicle is tracked based on the detection results (such as vehicle information like vehicle color and vehicle type). Since the vehicle information of the partial body images is incomplete, directly detecting the partial body images may lead to inaccurate detection results, thus affecting the vehicle tracking effect. Summary of the Invention
[0003] In view of the above, it is necessary to provide an image processing method, an in-vehicle device, and a storage medium, which can solve the technical problem that directly detecting partial body images with missing vehicle information leads to inaccurate vehicle detection results, thus affecting the vehicle tracking effect and further affecting driving safety.
[0004] On the one hand, this application provides an image processing method, and the method includes: obtaining multiple partial body images of an external vehicle, where the multiple partial body images include an upper body image obtained by a first photographing device and a lower body image obtained by a second photographing device, the first photographing angle of the first photographing device and the second photographing angle of the second photographing device are different, determining a target visual plane according to the photographing fields of the first photographing device and the second photographing device, obtaining the spatial coordinates of multiple vertices of the target visual plane, projecting each spatial coordinate into the coordinate system of the upper body image to obtain a first projection coordinate corresponding to each spatial coordinate, and projecting each spatial coordinate into the coordinate system of the lower body image to obtain a second projection coordinate corresponding to each spatial coordinate, generating a first transformation matrix corresponding to the upper body image or generating a second transformation matrix corresponding to the lower body image according to the first projection coordinate and the second projection coordinate corresponding to each spatial coordinate, transforming the upper body image according to the first transformation matrix to obtain a first transformed image corresponding to the upper body image, or transforming the lower body image according to the second transformation matrix to obtain a second transformed image corresponding to the lower body image, and generating a complete body image of the external vehicle according to the first transformed image and the lower body image, or according to the second transformed image and the upper body image.
[0005] In some embodiments of the present application, the target visual plane is in the overlapping spatial region between the first shooting field of view of the first shooting device corresponding to the upper body image and the second shooting field of view of the second shooting device corresponding to the lower body image.
[0006] In some embodiments of the present application, the step of projecting each spatial coordinate into the coordinate system of the upper body image to obtain the first projection coordinate corresponding to each spatial coordinate includes: projecting each spatial coordinate into the coordinate system of the upper body image by an orthogonal projection method or a perspective projection method to obtain the first projection coordinate corresponding to each spatial coordinate.
[0007] In some embodiments of the present application, the step of generating the first transformation matrix corresponding to the upper body image according to the first projection coordinate and the second projection coordinate corresponding to each spatial coordinate includes: multiplying the first homogeneous coordinate vector of each first projection coordinate by a preset parameter matrix to obtain the multiplication vector of each first projection coordinate, and generating the first transformation matrix according to the multiplication vector and the second homogeneous coordinate vector of the second projection coordinate corresponding to the multiplication vector.
[0008] In some embodiments of the present application, the preset parameter matrix includes multiple parameters. The step of generating the first transformation matrix according to the multiplication vector and the second homogeneous coordinate vector of the second projection coordinate corresponding to the multiplication vector includes: establishing an equation relationship between the multiplication vector of each first projection coordinate and the second homogeneous coordinate vector of the corresponding second projection coordinate to obtain multiple equations corresponding to the multiple parameters, solving all the parameter equations corresponding to the multiple first projection coordinates to obtain the parameter value corresponding to each parameter, and replacing each parameter in the preset parameter matrix with the corresponding parameter value to obtain the first transformation matrix.
[0009] In some embodiments of the present application, the step of transforming the upper body image according to the first transformation matrix to obtain the first transformed image corresponding to the upper body image includes: multiplying the homogeneous coordinate vector corresponding to the initial coordinate of each pixel point in the upper body image by the first transformation matrix to obtain the target coordinate corresponding to each initial coordinate, and replacing the pixel value of the pixel point at each target coordinate in the upper body image with the pixel value of the pixel point at the corresponding initial coordinate to obtain the first transformed image.
[0010] In some embodiments of the present application, the step of generating the complete body image of the external vehicle according to the first transformed image and the lower body image includes: cropping the body overlapping region between the first transformed image and the lower body image in the first transformed image to obtain a cropped image, and splicing the cropped image and the lower body image to obtain the complete body image.
[0011] In some embodiments of the present application, the method further includes: calling a target detection model to detect the complete vehicle body image, and obtaining the body coordinates of the external vehicle in the complete vehicle body image and the vehicle information of the external vehicle.
[0012] On the other hand, the present application provides a vehicle-mounted device, which includes: a memory storing at least one instruction; and a processor configured to obtain the instruction stored in the memory to implement the image processing method described above.
[0013] On the other hand, the present application provides a computer-readable storage medium storing at least one instruction, and the at least one instruction is obtained by a processor in a vehicle-mounted device to implement the image processing method.
[0014] Through the above embodiments, the upper body image and the lower body image are captured by different imaging devices. Since different imaging devices have different shooting perspectives, a transformation matrix for converting the shooting perspective of the first imaging device to that of the second imaging device, or a transformation matrix for converting the shooting perspective of the second imaging device to that of the first imaging device can be generated according to the first projection coordinates corresponding to multiple spatial coordinates. By transforming the upper body image with the first transformation matrix, the converted first transformed image can be made equivalent to the upper body image obtained by using the second imaging device for extended shooting. Or, by transforming the lower body image with the second transformation matrix, the converted second transformed image can be made equivalent to the lower body image obtained by using the first imaging device for extended shooting. Therefore, a complete vehicle body image of the external vehicle captured from the second shooting perspective can be generated according to the first transformed image and the lower body image, or a complete vehicle body image of the external vehicle captured from the first shooting perspective can be generated according to the second transformed image and the upper body image. Detecting the complete vehicle body image of the external vehicle can ensure the accuracy of the detection result and the vehicle tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a schematic structural diagram of a vehicle-mounted device provided by an embodiment of the present application.
[0016] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application.
[0017] Figure 3 is a schematic diagram of an upper body image provided by an embodiment of the present application.
[0018] Figure 4 is a schematic diagram of a lower body image provided by an embodiment of the present application.
[0019] Figure 5 It is a schematic diagram of a first projection plane provided by an embodiment of the present application.
[0020] Figure 6 It is a schematic diagram of a second projection plane provided by an embodiment of the present application.
[0021] Figure 7 It is a schematic diagram of a cropped image provided by an embodiment of the present application.
[0022] Figure 8 It is a schematic diagram of a complete vehicle body image provided by an embodiment of the present application.
[0023] Figure 9 It is a schematic diagram of a complete vehicle body area provided by an embodiment of the present application.
[0024] Figure 10 It is a flowchart of a calculation method for a first transformation matrix provided by an embodiment of the present application. Detailed implementation manners
[0025] It should be noted that, in the present application, "at least one" means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and drawings of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.
[0027] During vehicle driving, for the sake of safe and intelligent driving, external vehicle images around the vehicle are often obtained through imaging devices installed on the vehicle. However, due to limitations such as the imaging position and imaging angle, the captured images may only contain partial vehicle bodies of external vehicles. In related technologies, the captured partial vehicle body images are often directly detected, and the external vehicles are tracked based on the detection results (such as vehicle information such as vehicle color and vehicle type). Since the vehicle information of the partial vehicle body images is incomplete, directly detecting the partial vehicle body images may result in inaccurate detection results, thereby affecting the vehicle tracking effect.
[0028] To solve the above technical problems, the present application provides an image processing method, an in-vehicle device, and a storage medium, which can generate a complete vehicle body image, thereby ensuring the accuracy of vehicle detection results and the vehicle tracking effect. The image processing method provided by the embodiments of the present application can be applied to one or more in-vehicle devices.
[0029] As Figure 1 shown, it is a schematic structural diagram of an in-vehicle device provided by an embodiment of the present application. The in-vehicle device 10 may be an in-vehicle device such as an electronic control unit (ECU) or a body control module (BCM) of a vehicle. The embodiments of the present application do not impose any restrictions on the specific type of the in-vehicle device.
[0030] As Figure 1 shown, the in-vehicle device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is respectively coupled to the communication module 101, the memory 102, and the input / output interface 104 through the bus 105.
[0031] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more of the solutions for wired communication such as a universal serial bus (USB), a controller area network bus (CAN), a local interconnect network bus (LIN), and a flexray bus. The wireless communication module may provide one or more of the solutions for wireless communication such as wireless fidelity (Wi-Fi), Bluetooth (BT), a mobile communication network, frequency modulation (FM), near field communication (NFC), and infrared technology (IR).
[0032] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103 and can be used to store the executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data, etc. The random access memory can include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.
[0033] The non-volatile memory can also store executable programs and store user and application data, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 103. The non-volatile memory can include disk storage devices and flash memory.
[0034] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include a plurality of instructions. When the plurality of instructions are executed by the processor 103, an image processing method executable on the vehicle-mounted device 10 can be implemented.
[0035] In other embodiments, as Figure 1 shown, the vehicle-mounted device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the vehicle-mounted device 10.
[0036] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.
[0037] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute the computer program stored in the memory 102 to implement the above-mentioned image processing method.
[0038] The input / output interface 104 is used to provide channels for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can input information or visualize information.
[0039] The bus 105 is at least used to provide a communication channel for mutual communication between the communication module 101, the memory 102, the processor 103, and the input / output interface 104 in the vehicle-mounted device 10.
[0040] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the vehicle-mounted device 10. In other embodiments of the present application, the vehicle-mounted device 10 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0041] In other embodiments of the present application, the vehicle-mounted device 10 may also be replaced with an electronic device that can implement vehicle control. The electronic device includes, but is not limited to: a mobile phone, a tablet computer, a laptop computer, a computer, etc. The present application does not limit the electronic device.
[0042] As Figure 2 shown, it is a flowchart of an image processing method provided by an embodiment of the present application. According to different requirements, the order of each step in this flowchart can be adjusted according to actual requirements, and some steps can be omitted. As an example, the method is applied to a vehicle-mounted device in a vehicle, such as Figure 1 the vehicle-mounted device 10 shown.
[0043] S11. Obtain multiple partial body images of an external vehicle, where the multiple partial body images include an upper body image obtained by using a first photographing device and a lower body image obtained by using a second photographing device.
[0044] In some embodiments of the present application, the vehicle to which the method is applied may be referred to as the host vehicle, and the vehicles around the host vehicle may be referred to as external vehicles. The present application places no restrictions on external vehicles. The first photographing device and the second photographing device may be on-vehicle cameras and on-vehicle cameras on both sides of the vehicle (for example, the above-mentioned host vehicle). The present application places no restrictions on the specific types of the first photographing device and the second photographing device. The first photographing angle of the first photographing device is different from the second photographing angle of the second photographing device, and the first photographing field of view of the first photographing device is different from the second photographing field of view of the second photographing device.
[0045] In some embodiments of the present application, the in-vehicle device may control the first photographing device to photograph the external vehicle at the first photographing angle to obtain an upper body image, and control the second photographing device to photograph the external vehicle at the second photographing angle to obtain a lower body image.
[0046] For example, as Figure 3 shown, it is a schematic diagram of an upper body image provided by an embodiment of the present application. Figure 3 The upper body image in is the upper body image obtained by the in-vehicle device controlling the in-vehicle camera on the right rear side of the vehicle to photograph the external vehicle forward. As Figure 4 shown, it is a schematic diagram of a lower body image provided by an embodiment of the present application. Figure 4 The lower body image in is the lower body image obtained by the in-vehicle device controlling the in-vehicle camera on the right front side of the vehicle to photograph backward.
[0047] S12. Determine a target visual plane according to the photographing fields of view of the first photographing device and the second photographing device, and obtain the spatial coordinates of multiple vertices of the target visual plane.
[0048] In some embodiments of the present application, the target visual plane may be in the overlapping spatial region between the first photographing field of view of the first photographing device corresponding to the upper body image and the second photographing field of view of the second photographing device corresponding to the lower body image. The shape of the target visual plane may be a rectangle. The above examples are not restricted in practical applications.
[0049] In some embodiments of the present application, the in-vehicle device may determine a rectangular plane as the target visual plane in the overlapping spatial region of the first shooting field of view and the second shooting field of view. The coordinates of multiple vertices of the target visual plane may be the spatial coordinates of the four vertices of the rectangular plane. For example, the target visual plane includes four vertices A, B, C, and D, and the spatial coordinates of the four vertices A, B, C, and D are A(1, 4.5, 0), B(1, 4.5, 1.6), C(-1, 4.5, 0), and D(-1, 4.5, 1.6) respectively.
[0050] S13. Project each spatial coordinate into the coordinate system of the upper body image to obtain the first projection coordinate corresponding to each spatial coordinate, and project each spatial coordinate into the coordinate system of the lower body image to obtain the second projection coordinate corresponding to each spatial coordinate.
[0051] In some embodiments of the present application, when the in-vehicle device projects each spatial coordinate into the coordinate system of the upper body image to obtain the first projection coordinate corresponding to each spatial coordinate, it includes: the in-vehicle device projects each spatial coordinate into the coordinate system of the upper body image through an orthogonal projection method or a perspective projection method to obtain the first projection coordinate corresponding to each spatial coordinate.
[0052] Among them, the coordinate system of the upper body image may be an image coordinate system or a pixel coordinate system, and the present application does not limit this. The first projection coordinate corresponding to each spatial coordinate in the pixel coordinate system of the upper body image is a two-dimensional coordinate.
[0053] For example, the in-vehicle device may use the geometric center point of the upper body image as the origin, the straight line passing through the origin and parallel to the first row of pixel points of the upper body image as the x-axis, and the straight line passing through the origin and parallel to the first column of pixel points of the upper body image as the y-axis to construct the image coordinate system of the upper body image. The in-vehicle device may use the pixel point at the first row and first column position in the upper body image as the origin, the first row of pixel points of the upper body image as the x-axis, and the first column of pixel points of the upper body image as the y-axis to construct the pixel coordinate system of the upper body image.
[0054] In other embodiments of the present application, the in-vehicle device may project each spatial coordinate into the coordinate system of the upper body image by other means, and the present application does not limit the projection method.
[0055] In some embodiments of the present application, the generation process of the second projection coordinate is basically the same as that of the first projection coordinate, so the present application will not repeat the description.
[0056] In this embodiment, multiple first projection coordinates may form the first projection plane corresponding to the target visual plane in the upper body image, and multiple second projection coordinates may form the second projection plane corresponding to the target visual plane in the lower body image. For example, as Figure 5As shown, it is a schematic diagram of the first projection plane provided by an embodiment of the present application. Figure 5 The rectangular area indicated by the dashed line in the figure is the first projection plane. As Figure 6 shown, it is a schematic diagram of the second projection plane provided by an embodiment of the present application. Figure 6 The rectangle indicated by the dashed line in the figure is the second projection plane. From Figure 5 and Figure 6 it can be seen that the position of the first projection plane in the vehicle body of the upper body image is the same as the position of the second projection plane in the vehicle body of the lower body image.
[0057] S14. Generate a first transformation matrix corresponding to the upper body image or a second transformation matrix corresponding to the lower body image according to the first projection coordinates and the second projection coordinates corresponding to each spatial coordinate.
[0058] In some embodiments of the present application, the vehicle-mounted device generating the first transformation matrix corresponding to the upper body image according to the first projection coordinates and the second projection coordinates corresponding to each spatial coordinate includes: the vehicle-mounted device multiplying the first homogeneous coordinate vector of each first projection coordinate by a preset parameter matrix to obtain a multiplication vector of each first projection coordinate, and generating a first transformation matrix according to the multiplication vector and the second homogeneous coordinate vector of the second projection coordinate corresponding to the multiplication vector.
[0059] Among them, the preset parameter matrix includes multiple parameters, and the preset parameter matrix can be a 3x3 matrix. For example, the preset parameter matrix can be The first homogeneous coordinate vector of each first projection coordinate is a vector generated according to the corresponding first projection coordinate and the value "1". For example, the first homogeneous coordinate vector corresponding to the first projection coordinate (u1, v1) is The generation process of the second homogeneous coordinate vector of each second projection coordinate is basically the same as the generation process of the first homogeneous coordinate vector, and the present application will not repeat the description here. The second projection coordinate corresponding to the multiplication vector of any first projection coordinate is the second projection coordinate generated by projecting the spatial coordinate corresponding to any first projection coordinate.
[0060] In this embodiment, the first transformation matrix is a matrix that converts the shooting perspective of the first shooting device to the shooting perspective of the second shooting device.
[0061] In other embodiments of the present application, the vehicle-mounted device generating the second transformation matrix corresponding to the lower body image according to the first projection coordinates and the second projection coordinates corresponding to each spatial coordinate includes: the vehicle-mounted device multiplying the second homogeneous coordinate vector of each second projection coordinate by a preset parameter matrix to obtain a product vector of each second projection coordinate, and generating a second transformation matrix according to the product vector and the first homogeneous coordinate vector of the first projection coordinate corresponding to the product vector.
[0062] Among them, the first projection coordinate corresponding to the product vector of any second projection coordinate is the first projection coordinate generated by projecting the spatial coordinate corresponding to any second projection coordinate. The generation process of the product vector is basically the same as the generation process of the multiplied vectors. The process of generating the second transformation matrix based on the product vector and the first homogeneous coordinate vector corresponding to the first projection coordinate of the product vector is basically the same as the process of generating the first transformation matrix based on the multiplied vectors and the second homogeneous coordinate vector corresponding to the second projection coordinate of the multiplied vectors. Therefore, this application will not repeat the description.
[0063] In this embodiment, the second transformation matrix is a matrix that transforms the shooting perspective of the second shooting device to the shooting perspective of the first shooting device.
[0064] S15. Transform the upper body image according to the first transformation matrix to obtain a first transformed image corresponding to the upper body image, or transform the lower body image according to the second transformation matrix to obtain a second transformed image corresponding to the lower body image.
[0065] In some embodiments of this application, the vehicle-mounted device transforms the upper body image according to the first transformation matrix to obtain a first transformed image corresponding to the upper body image, which includes: the vehicle-mounted device multiplies the homogeneous coordinate vector corresponding to the first initial coordinate of each pixel point in the upper body image by the first transformation matrix to obtain the first target coordinate corresponding to each first initial coordinate, and in the upper body image, replaces the pixel value of the pixel point at each first target coordinate with the pixel value of the pixel point at the corresponding first initial coordinate to obtain the first transformed image.
[0066] Among them, the generation method of the homogeneous coordinate vector corresponding to each first initial coordinate can refer to the generation method of the first homogeneous coordinate vector in the above text.
[0067] In this embodiment, by transforming the upper body image with the first transformation matrix, the obtained first transformed image can be equivalent to the upper body image obtained by using the second shooting device for extended shooting with the second shooting device.
[0068] In other embodiments of this application, the vehicle-mounted device transforms the lower body image according to the second transformation matrix to obtain a second transformed image corresponding to the lower body image, which includes: the vehicle-mounted device multiplies the homogeneous coordinate vector corresponding to the second initial coordinate of each pixel point in the lower body image by the second transformation matrix to obtain the second target coordinate corresponding to each second initial coordinate, and in the lower body image, replaces the pixel value of the pixel point at each second target coordinate with the pixel value of the pixel point at the corresponding second initial coordinate to obtain the second transformed image.
[0069] In this embodiment, by using the second conversion matrix to convert the lower body image, the converted second conversion image can be made equivalent to the lower body image obtained by extending the first shooting device with the first shooting device.
[0070] S16. Generate a complete body image of the external vehicle according to the first conversion image and the lower body image, or according to the second conversion image and the upper body image.
[0071] In some embodiments of the present application, the vehicle-mounted device generating a complete body image of the external vehicle according to the first conversion image and the lower body image includes: the vehicle-mounted device cuts out the first body coincidence area between the first conversion image and the lower body image in the first conversion image to obtain a cut image, and splices the cut image and the lower body image to obtain a complete body image.
[0072] Wherein, the vehicle-mounted device can detect the first body coincidence area between the first conversion image and the lower body image through a preset algorithm. For example, the preset algorithm can be an edge detection algorithm.
[0073] In this embodiment, a complete body image of the external vehicle taken from the second shooting perspective can be generated according to the first conversion image and the lower body image.
[0074] In other embodiments of the present application, the vehicle-mounted device generating a complete body image of the external vehicle according to the second conversion image and the upper body image includes: the vehicle-mounted device cuts out the second body coincidence area between the second conversion image and the upper body image in the second conversion image to obtain a cut image, and splices the cut image and the upper body image to obtain a complete body image.
[0075] Among them, the generation process of the second body coincidence area is basically the same as that of the first body coincidence area, so the present application will not repeat the description here.
[0076] In this embodiment, a complete body image of the external vehicle taken from the first shooting perspective can be generated according to the second conversion image and the upper body image.
[0077] For example, as Figure 7 shown, it is a schematic diagram of a cut image provided by an embodiment of the present application. Figure 7 It is the cut image obtained after cutting out the first body coincidence area between the first conversion image and the lower body image in the first conversion image. For example, as Figure 8 shown, it is a schematic diagram of a complete body image provided by an embodiment of the present application. Figure 8 It is for Figure 7 the complete body image obtained by splicing the cut image in Figure 8It can be seen that the complete body image includes the complete body of the external vehicle.
[0078] In some embodiments of the present application, after obtaining the complete body image, the in-vehicle device may call a target detection model to detect the complete body image, and obtain the body coordinates of the external vehicle in the complete body image and the vehicle information of the external vehicle.
[0079] Among them, the target detection model can be set by itself, and the present application does not limit this. For example, the target detection model can be a convolutional neural network model (Convolutional Neural Network, CNN) with a VGG network as the backbone network. The target detection model can be obtained through pre-training. The vehicle information includes, but is not limited to: vehicle type, vehicle color, vehicle size, etc.
[0080] After obtaining the body coordinates, the in-vehicle device can frame the complete body area of the external vehicle in the complete body image according to the body coordinates, extract features from the body area to obtain vehicle features, and identify the vehicle features to obtain the vehicle information of the external vehicle. For example, as Figure 9 shown, it is a schematic diagram of the complete body area provided by an embodiment of the present application. In Figure 9 it, the rectangular area represented by the dotted line is the complete body area, and the complete body of the external vehicle is included inside the complete body area.
[0081] In this embodiment, detecting the complete body image of the external vehicle can ensure the accuracy of detection results such as vehicle information and the vehicle tracking effect.
[0082] Through the above embodiments, the upper body image and the lower body image are obtained by different photographing devices. Since different photographing devices have different photographing perspectives, a transformation matrix for transforming the photographing perspective of the first photographing device to the photographing perspective of the second photographing device, or a transformation matrix for transforming the photographing perspective of the second photographing device to the photographing perspective of the first photographing device can be generated according to the first projection coordinates corresponding to multiple spatial coordinates. By transforming the upper body image through the first transformation matrix, the transformed first transformed image can be made equivalent to the upper body image obtained by the second photographing device extending with the second photographing device. Or, by transforming the lower body image through the second transformation matrix, the transformed second transformed image can be made equivalent to the lower body image obtained by the first photographing device extending with the first photographing device. Therefore, a complete body image of the external vehicle obtained by photographing from the second photographing perspective can be generated according to the first transformed image and the lower body image. Or, a complete body image of the external vehicle obtained by photographing from the first photographing perspective can be generated according to the second transformed image and the upper body image. Detecting the complete body image of the external vehicle can ensure the accuracy of the detection result and the vehicle tracking effect.
[0083] In some embodiments of the present application, as Figure 10 shown, it is a flowchart of a calculation method of a first transformation matrix provided by an embodiment of the present application, which specifically includes the following steps:
[0084] S151, establish an equation relationship between the multiplication vector of each first projection coordinate and the second homogeneous coordinate vector of the corresponding second projection coordinate to obtain a vector equation, and simplify the vector equation to obtain multiple parameter equations corresponding to multiple parameters.
[0085] In some embodiments of the present application, each first projection coordinate corresponds to multiple parameter equations.
[0086] For example, continuing with the above embodiments, if the preset parameter matrix is The first homogeneous coordinate vector of any first projection coordinate is The second homogeneous coordinate vector of the second projection coordinate corresponding to the any first projection coordinate is Preset parameter matrix First homogeneous coordinate vector And the second homogeneous coordinate vector The corresponding vector equation can refer to formula (1), and the multiple parameter equations obtained by simplifying formula (1) can refer to formulas (2)-(4):
[0087]
[0088] u2 = au1 + bv1 + c; (2)
[0089] v2 = du1 + ev1 + f; (3)
[0090] 1 = gu1 + hv1 + i. (4)
[0091] S152. Solve all the parameter equations corresponding to multiple first projection coordinates to obtain the parameter values corresponding to each parameter.
[0092] In some embodiments of the present application, since each first projection coordinate corresponds to multiple parameter equations, all the parameter equations obtained from multiple first projection coordinates can be solved to obtain the parameter values corresponding to each parameter.
[0093] S153. Replace each parameter in the preset parameter matrix with the corresponding parameter value to obtain the first conversion matrix.
[0094] In this embodiment, each parameter has a corresponding parameter value. By using each parameter value to replace the corresponding parameter in the preset parameter matrix, the first conversion matrix can be obtained. When the matrix dimension of the preset parameter matrix is 3x3, the matrix dimension of the first conversion matrix is also 3x3.
[0095] The embodiments of the present application also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and the computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above various embodiments of the present application.
[0096] Among them, the computer-readable storage medium can be the internal memory of the vehicle-mounted device described in the above embodiments, such as the hard disk or memory of the vehicle-mounted device. The computer-readable storage medium can also be an external storage device of the vehicle-mounted device, such as a plug-in hard disk equipped on the vehicle-mounted device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc.
[0097] In some embodiments, the computer-readable storage medium may include a storage program area and a storage data area. Among them, the storage program area can store an operating system, application programs required for at least one function, etc.; the storage data area can store data created according to the use of the vehicle-mounted device.
[0098] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0099] In several embodiments provided by this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0100] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in each embodiment of this application, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of a combination of hardware and software functional modules.
[0101] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of this application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in this application. Any associated drawing marks in the claims should not be regarded as limiting the claims involved. In addition, obviously, the word "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in this application can also be implemented by one unit or device through software or hardware. Words such as first and second are used to represent names and do not indicate any specific order.
[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of this application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of this application.
Claims
1. An image processing method, applied to an in-vehicle device, characterized in that, The method includes: Obtaining multiple partial body images of an external vehicle, where the multiple partial body images include an upper body image obtained by a first imaging device and a lower body image obtained by a second imaging device, and the first imaging angle of the first imaging device and the second imaging angle of the second imaging device are different; Determining a target visual plane according to the imaging fields of view of the first imaging device and the second imaging device, and obtaining the spatial coordinates of multiple vertices of the target visual plane; Projecting each spatial coordinate into the coordinate system of the upper body image to obtain a first projection coordinate corresponding to each spatial coordinate, and projecting each spatial coordinate into the coordinate system of the lower body image to obtain a second projection coordinate corresponding to each spatial coordinate; Generating a first transformation matrix corresponding to the upper body image according to the first projection coordinate and the second projection coordinate corresponding to each spatial coordinate, or generating a second transformation matrix corresponding to the lower body image; Transforming the upper body image according to the first transformation matrix to obtain a first transformed image corresponding to the upper body image, or transforming the lower body image according to the second transformation matrix to obtain a second transformed image corresponding to the lower body image; Generating a complete body image of the external vehicle according to the first transformed image and the lower body image, or according to the second transformed image and the upper body image.
2. The image processing method according to claim 1, wherein The target visual plane is in the overlapping spatial region between the first imaging field of view of the first imaging device corresponding to the upper body image and the second imaging field of view of the second imaging device corresponding to the lower body image.
3. The image processing method according to claim 1, characterized in that The step of projecting each spatial coordinate into the coordinate system of the upper body image to obtain a first projection coordinate corresponding to each spatial coordinate includes: Projecting each spatial coordinate into the coordinate system of the upper body image by an orthogonal projection method or a perspective projection method to obtain a first projection coordinate corresponding to each spatial coordinate.
4. The image processing method according to claim 1, characterized in that The step of generating a first transformation matrix corresponding to the upper body image according to the first projection coordinate and the second projection coordinate corresponding to each spatial coordinate includes: Multiplying the first homogeneous coordinate vector of each first projection coordinate by a preset parameter matrix to obtain a multiplied vector of each first projection coordinate; Generating a first transformation matrix according to the multiplied vector and the second homogeneous coordinate vector of the second projection coordinate corresponding to the multiplied vector.
5. The image processing method according to claim 4, characterized in that The preset parameter matrix includes multiple parameters, and the step of generating a first transformation matrix according to the multiplied vector and the second homogeneous coordinate vector of the second projection coordinate corresponding to the multiplied vector includes: Establishing an equation relationship between the multiplied vector of each first projection coordinate and the second homogeneous coordinate vector of the corresponding second projection coordinate to obtain a vector equation, and simplifying the vector equation to obtain multiple parameter equations corresponding to the multiple parameters; Solving all the parameter equations corresponding to multiple first projection coordinates to obtain the parameter values corresponding to each parameter; Replacing each parameter in the preset parameter matrix with the corresponding parameter value to obtain the first transformation matrix.
6. The image processing method according to claim 1, wherein, Converting the upper body image according to the first conversion matrix to obtain a first converted image corresponding to the upper body image includes: Multiplying the homogeneous coordinate vector corresponding to the initial coordinate of each pixel point in the upper body image by the first conversion matrix to obtain a target coordinate corresponding to each initial coordinate; Replacing the pixel value of the pixel point at each target coordinate in the upper body image with the pixel value of the pixel point at the corresponding initial coordinate to obtain the first converted image.
7. The image processing method according to claim 1, characterized in that Generating a complete body image of the external vehicle according to the first converted image and the lower body image includes: Cropping a body overlapping region between the first converted image and the lower body image in the first converted image to obtain a cropped image; Stitching the cropped image and the lower body image to obtain the complete body image.
8. The image processing method according to claim 1 or 7, characterized in that, The method further includes: Invoking a target detection model to detect the complete body image to obtain the body coordinates of the external vehicle in the complete body image and the vehicle information of the external vehicle.
9. A vehicle-mounted device, characterized in that, The in-vehicle device includes: A memory storing at least one instruction; and A processor that executes the at least one instruction to implement the image processing method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: At least one instruction is stored in the computer-readable storage medium, and when the at least one instruction is executed by a processor in an in-vehicle device, the image processing method according to any one of claims 1 to 8 is implemented.