Vehicle-mounted panoramic monitoring system linked with virtual reality device and vehicle

Abstract

The invention provides a vehicle-mounted panoramic monitoring system linked with a virtual reality device and a vehicle, and belongs to the technical field of virtual reality. The system comprises a pan-tilt camera used for transmitting an original image sequence acquired by performing panoramic scanning on an external scene of a vehicle body to a vehicle-mounted processor; the vehicle-mounted processor is used for synchronizing a first panoramic video frame obtained by performing three-dimensional scene reconstruction by using the original image sequence to each virtual reality device; and the virtual reality device is used for adjusting the first panoramic video frame into a second panoramic video frame corresponding to a fixed view field of a user of the virtual reality device in the vehicle body, and performing near-to-eye display on the second panoramic video frame. According to the vehicle-mounted panoramic monitoring system linked with the virtual reality device and the vehicle provided by the invention, the panoramic video stream is always kept consistent with the orientation of the user in the moving process of the vehicle, so that a viewer has a scene feeling and a sense of immersion.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a vehicle-mounted panoramic monitoring system and a vehicle linked with a virtual reality device. Background Art

[0002] Passengers of vehicles or other transportation vehicles in a restricted field of vision or in a sealed environment need to observe not only the situation inside the vehicle, but also the situation outside the vehicle, so as to inform the crew of the situation outside the vehicle or quickly get to work after getting off the vehicle.

[0003] The existing system mainly uses a camera with a pan-tilt system and an in-vehicle display system. A control device is required to rotate the pan-tilt system to adjust the viewing angle. When there are many users, they can only operate one by one in sequence. In addition, the panoramic video stream collected by the camera is displayed on the in-vehicle display, which is very different from the human eye perception, making it difficult for viewers to have a sense of presence. Therefore, the existing in-vehicle panoramic monitoring display solution has a relatively low sense of presence and currentness. Summary of the invention

[0004] The present invention provides a vehicle-mounted panoramic monitoring system and a vehicle linked with a virtual reality device, so as to solve the defects of the prior art in-vehicle panoramic monitoring display scheme that the sense of presence and the timeliness are both relatively low.

[0005] The present invention provides a vehicle-mounted panoramic monitoring system linked with a virtual reality device, comprising a pan / tilt camera and a vehicle-mounted processor arranged on a vehicle body, and a plurality of virtual reality devices communicatively connected to the vehicle-mounted processor; The pan / tilt camera is used for panoramic scanning of the external scene of the vehicle body and transmitting the acquired original image sequence to the vehicle-mounted processor; The vehicle-mounted processor is used to synchronize the first panoramic video frame obtained by reconstructing the three-dimensional scene using the original image sequence to each of the virtual reality devices; The virtual reality device is used to adjust the first panoramic video frame to a second panoramic video frame corresponding to a fixed field of view of a user of the virtual reality device in the vehicle body, and to perform near-eye display of the second panoramic video frame; The fixed field of view is determined after aligning the first panoramic video frame with a polar coordinate system, and the polar coordinate system is constructed with the front of the vehicle as a direction of 0 degrees.

[0006] According to the vehicle-mounted panoramic monitoring system linked with a virtual reality device provided by the present invention, an inertial navigation device is arranged on the virtual reality device, and the inertial navigation device is aligned with the polar coordinate system; The virtual reality device is further used to collect head posture information according to the inertial navigation device, adjust the second panoramic video frame to a third panoramic video frame according to the head posture information, and perform near-eye display of the third panoramic video frame.

[0007] According to a vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by the present invention, the near-eye display of the third panoramic video frame includes: Determine a first viewing angle range and a second viewing angle range according to the head posture information; Using the first viewing angle range to extract a first sub-video frame from the third panoramic video frame, using the second viewing angle range to extract a second sub-video frame from the third panoramic video frame, and alternately displaying the first sub-video frame and the second sub-video frame in the left image field and the right image field of the virtual reality device; The first viewing angle range is determined based on the left eye field of view of the user of the virtual reality device under the current head posture; the second viewing angle range is determined based on the right eye field of view of the user of the virtual reality device under the current head posture.

[0008] According to the present invention, a vehicle-mounted panoramic monitoring system linked with a virtual reality device is provided, wherein the virtual reality device is provided with an eye tracker; The virtual reality device is further used to collect eye movement information according to the eye tracker, adjust the second panoramic video frame to a fourth panoramic video frame according to the eye movement information, and perform near-eye display of the fourth panoramic video frame.

[0009] According to a vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by the present invention, the near-eye display of the fourth panoramic video frame includes: Determining a third viewing angle range and a fourth viewing angle range according to the eye movement information; Using the third viewing angle range to extract a third sub-video frame from the third panoramic video frame, using the fourth viewing angle range to extract a fourth sub-video frame from the third panoramic video frame, and alternately displaying the third sub-video frame and the fourth sub-video frame in the left image field and the right image field of the virtual reality device; Among them, the third viewing angle range is determined based on the left eye field of view of the user of the virtual reality device at the current eye position; the fourth viewing angle range is determined based on the right eye field of view of the user of the virtual reality device at the current eye position.

[0010] According to a vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by the present invention, the vehicle-mounted processor performs the following steps when the vehicle-mounted processor uses the original image sequence to reconstruct the first panoramic video frame obtained by the three-dimensional scene: Acquire a registration pixel pair based on a field of view overlap area between a first original image frame and a second original image frame in the original image sequence; Using the registered pixel pairs, the first original image frame and the second original image frame are fused frame by frame until the second original image frame is the last original image frame in the original image sequence, thereby obtaining the first panoramic video frame; The first original image frame and the second original image frame are any two adjacent image frames in the original image sequence.

[0011] According to a vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by the present invention, the method of obtaining a registration pixel pair based on a field of view overlap area between a first original image frame and a second original image frame in the original image sequence includes: Determining an optimal registration line pixel position based on the first original image frame and the second original image frame; Based on the optimal registration line pixel position, the field of view overlap area between the first original image frame and the second original image frame is determined, so that the on-board processor can extract the registration pixel pairs from the first original image frame and the second original image frame according to the field of view overlap area.

[0012] According to a vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by the present invention, the first original image frame and the second original image frame are fused frame by frame by using the registered pixel pair until the second original image frame is the last original image frame in the original image sequence to obtain the first panoramic video frame, including: fusing the first original image frame and the second original image frame using the registered pixel pairs to obtain a first fused image frame; Updating the first original image frame to the first fused image frame, and updating the second original image frame to the second original image frame to be fused after the next image frame in the original image sequence, until the second original image frame is the last original image frame in the original image sequence, to obtain a second fused image frame; The second fused image frame is straightened based on the pixel position of the best registration line to determine the first panoramic video frame.

[0013] According to a vehicle-mounted panoramic monitoring system linked with a virtual reality device provided by the present invention, the pan-tilt camera includes a pan-tilt stabilizer and a camera module arranged on the pan-tilt stabilizer; The gimbal stabilizer drives the camera module to rotate around the rotation axis, so that the camera module can collect an original image sequence corresponding to the rotation axis; The vehicle-mounted processor is further used to fuse original image sequences corresponding to different rotation axes; Wherein, the rotation axis includes at least one of a pitch axis, a yaw axis and a roll axis.

[0014] The present invention also provides a vehicle, comprising a vehicle body, and a vehicle-mounted panoramic monitoring system as described in any of the above items and linked with a virtual reality device, which is loaded in the vehicle body.

[0015] The vehicle-mounted panoramic monitoring system and vehicle linked with a virtual reality device provided by the present invention sense the external scene of the vehicle through a pan-tilt camera, and the vehicle-mounted processor reconstructs the three-dimensional scene of the sensed original image sequence, and synchronizes the obtained first panoramic video frame in the form of a video stream to the virtual reality device worn by each user in the vehicle, and then the virtual reality device aligns the first panoramic video frame with the position of the user in the vehicle, and then displays the obtained second panoramic video frame near the eye. The panoramic video stream is always kept consistent with the user's position during the movement of the vehicle, greatly reducing the difference in human eye perception, and giving the viewer a sense of presence and immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a structural schematic diagram of a vehicle-mounted panoramic monitoring system linked with a virtual reality device provided by the present invention; Figure 2 It is a structural schematic diagram of a vehicle provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0020] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0021] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] Figure 1 Schematic diagram of the structure of the vehicle-mounted panoramic monitoring system linked with the virtual reality device provided by the present invention. Figure 1 As shown, the vehicle-mounted panoramic monitoring system linked to a virtual reality device provided by an embodiment of the present invention includes a pan-tilt camera 110 and a vehicle-mounted processor 120 arranged on a vehicle body, and a plurality of virtual reality devices 130 communicatively connected to the vehicle-mounted processor 120.

[0023] The pan-tilt camera 110 is used for panoramic scanning of the external scene of the vehicle body and transmitting the acquired original image sequence to the vehicle-mounted processor 120 .

[0024] The vehicle-mounted processor 120 is used to synchronize the first panoramic video frame obtained by reconstructing the three-dimensional scene using the original image sequence to each of the virtual reality devices 130 .

[0025] The virtual reality device 130 is used to adjust the first panoramic video frame to a second panoramic video frame corresponding to a fixed field of view of a user of the virtual reality device 130 in the vehicle body, and perform near-eye display of the second panoramic video frame.

[0026] The fixed field of view is determined after aligning the first panoramic video frame with a polar coordinate system, and the polar coordinate system is constructed with the front of the vehicle as a direction of 0 degrees.

[0027] Specifically, the vehicle-mounted panoramic monitoring system linked with the virtual reality device includes two parts: a hardware system and a software system.

[0028] Among them, the hardware system is mainly composed of fixed devices such as wire-controlled brakes and wire-controlled throttle carried by the vehicle body itself, as well as sensing elements such as a pan-tilt camera 110 and a millimeter-wave radar that are flexibly installed on the vehicle body according to actual needs. During the movement of the vehicle, the pan-tilt camera 110 is used to perceive the external environment from multiple angles in real time, and the images of the surrounding scenes outside the vehicle body taken successively in each working cycle are stored in the original image sequence in sequence, and the obtained original image sequence is transmitted to the background on-board processor 120.

[0029] The software system is mainly configured in the on-board processor 120 to provide users with a human-computer interaction interface, image processing algorithm functions, etc. The on-board processor 120 uses the built-in image processing algorithm to stitch and fuse the original image sequence to reconstruct the first panoramic video frame corresponding to the three-dimensional scene, and then aligns it with the vehicle coordinate system with the front of the vehicle as the 0 degree direction, and publishes it as a video stream.

[0030] The embodiment of the present invention does not specifically limit the publishing platform of the video stream.

[0031] Exemplarily, the on-board processor 120 can use the human-computer interaction interface as a publishing platform for the video stream. The human-computer interaction interface is oriented to users in the cockpit of the vehicle body (for example, the driver or the co-pilot), so that the users in the cockpit can learn about the situation outside the vehicle through the human-computer interaction interface.

[0032] For example, the vehicle-mounted processor 120 may use the virtual reality device 130 as a publishing platform for video streams, so that a user wearing the virtual reality device 130 in the vehicle can obtain the situation outside the vehicle through the near-eye display of the virtual reality device 130 .

[0033] When the user wears the virtual reality device 130 that is matched with the vehicle body and activates it, the virtual reality device 130 establishes a communication connection with the vehicle processor 120 to receive the first panoramic video frame sent by the vehicle processor 120. Since different users have different positions in the vehicle body, it is necessary to associate the position of the corresponding user in the vehicle with the position in the actual geographical environment through the relative position relationship between the center of mass of the user and the vehicle body, obtain a fixed field of view corresponding to the associated position, and align the visual center of the first panoramic video frame with the center of the fixed field of view. After obtaining the second panoramic video frame that matches the actual position of the user, it is published in the near-eye display of the virtual reality device 130 in the form of a video stream.

[0034] For example, first, a panoramic video is generated by real-time video processing and displayed on a spherical screen. The VR helmet is positioned in the vehicle's cab. When the wearer turns his head, the posture of the VR helmet is captured to determine the area in the spherical screen corresponding to the viewing angle. Based on this area, a video stream is generated in the VR glasses, allowing the wearer to observe the scene outside the vehicle without leaving the vehicle's cab.

[0035] The embodiment of the present invention senses the external scene of the vehicle through the pan-tilt camera, and the vehicle-mounted processor reconstructs the three-dimensional scene of the sensed original image sequence, and synchronizes the obtained first panoramic video frame in the form of a video stream to the virtual reality device worn by each user in the vehicle, and then the virtual reality device aligns the first panoramic video frame with the position of the user in the vehicle, and then displays the obtained second panoramic video frame near the eye. The panoramic video stream is always kept consistent with the user's position during the movement of the vehicle, greatly reducing the difference in human eye perception, and giving the viewer a sense of presence and immersion.

[0036] Based on any of the above embodiments, the virtual reality device 130 is provided with an inertial navigation device.

[0037] The virtual reality device 130 is further used to collect head posture information according to the inertial navigation device, adjust the second panoramic video frame to a third panoramic video frame according to the head posture information, and perform near-eye display on the third panoramic video frame.

[0038] Specifically, an inertial navigation device is set on the main surface of the virtual reality device 130 so that the fixed field of view changes accordingly when the user's line of sight is aimed at other directions, and the user's head posture information is obtained through the inertial navigation device so that the virtual reality device 130 can adjust the visual center of the second panoramic video frame accordingly, and after obtaining the third panoramic video frame that matches the user's current alignment direction within its fixed field of view, it is published in the form of a video stream on the near-eye display of the virtual reality device 130.

[0039] The inertial navigation device includes but is not limited to an inertial sensor (IMU), an acceleration sensor, a gyroscope, etc., to measure the three-axis angular velocity, acceleration, and magnetic declination of the user's head in the user coordinate system.

[0040] The embodiment of the present invention extracts the head posture information collected by the user of the virtual reality device through an inertial navigation device, determines the user's focus area in a fixed field of view, and aligns the visual center of the panoramic video frame with the focus area. Adaptive adjustment of the panoramic rendering content is achieved according to the head posture, so that when the user's head moves or rotates, the user can sense the 3D rotation of the user by using an external device and compensate for the delay, which greatly reduces the dizziness caused by display lag.

[0041] Based on any of the above embodiments, the near-eye display of the third panoramic video frame includes: determining a first viewing angle range and a second viewing angle range according to the head posture information.

[0042] The first viewing angle range is determined according to the left eye field of view of the user of the virtual reality device in the current head posture, and the second viewing angle range is determined according to the right eye field of view of the user of the virtual reality device in the current head posture.

[0043] Specifically, the virtual reality device 130 uses the head posture information to calculate a first viewing angle range centered on the left eye's sight line under the current head posture, and a second viewing angle range centered on the right eye's sight line under the current head posture in the user coordinate system.

[0044] A first sub-video frame is extracted from the third panoramic video frame using the first viewing angle range, and a second sub-video frame is extracted from the third panoramic video frame using the second viewing angle range. The first sub-video frame and the second sub-video frame are alternately displayed in the left field and the right field of the virtual reality device.

[0045] Specifically, the virtual reality device 130 uses the first viewing angle range to capture a first sub-video frame with the left eye as the main visual center from the third panoramic video frame of 360°, and uses the second viewing angle range to capture a second sub-video frame with the right eye as the main visual center from the third panoramic video frame of 360°, and sends a stereo signal (left eye->right eye->left eye->right eye->in sequence and continuously alternately) to the screen of the near-eye display through the control IC, and simultaneously sends a synchronization signal to the virtual reality device 130 to synchronize the left and right eye images, so that the left eye sees the scene that the left eye should see (i.e., the first sub-video frame), and the right eye sees the scene that the right eye should see (i.e., the first sub-video frame).

[0046] The embodiment of the present invention determines the first viewing angle range and the second viewing angle range corresponding to the left and right eyes respectively according to the head posture information, and uses the first viewing angle range and the second viewing angle range to intercept the first sub-video frame and the second sub-video frame from the third panoramic video frame, so that the left and right eye images are continuously and alternately displayed on the screen. Combined with the physiological characteristics of the visual persistence of the human eye, it can simulate a real stereoscopic 3D image.

[0047] Based on any of the above embodiments, an eye tracker is disposed in the virtual reality device 130 .

[0048] The virtual reality device 130 is further used to collect eye movement information according to the eye tracker, adjust the second panoramic video frame to a fourth panoramic video frame according to the eye movement information, and perform near-eye display on the fourth panoramic video frame.

[0049] Specifically, an eye tracker is set in the near-eye display of the virtual reality device 130 towards the user, so that the eye tracker can be used to capture the movement trajectory of the eyeball when the user's line of sight is aimed at other directions, and the user's eye movement information is obtained through the eye tracker, so that the virtual reality device 130 can adjust the visual center of the second panoramic video frame accordingly, and after obtaining the fourth panoramic video frame that matches the position of the user's eyeball in its fixed field of view, it is published in the near-eye display of the virtual reality device 130 in the form of a video stream.

[0050] Among them, the working principle of the eye tracker includes but is not limited to mechanical recording method, optical recording method, modern eye tracking recording method, etc., to measure the user's eye movement indicators, mainly including: Fixation: Definitely the most prominent metric in the eye tracking literature. Fixation constitutes the basic unit of measurement, a fixation is equal to one raw sample captured by the eye tracker.

[0051] Fixation: If the gaze point is maintained for a certain amount of time, it becomes a fixation, during which time the eyes will be locked on a specific object.

[0052] Region of Interest (AOI): In order to focus the analysis on a specific area of ​​the stimulus, an AOI can be defined and, if desired, compared to other AOIs.

[0053] Heatmaps: Heatmaps are visualizations of fixation positions over time as an overlay of specific stimuli, which can be aggregated for comparisons across groups, or across participants.

[0054] Fixation Order: By detailing the order and direction in which fixations occur, it is possible to see what first captures the participant's attention and the way in which the stimuli are viewed.

[0055] Time to First Fixation (TTFF): Similar to the above, the time between stimulus onset and seeing a defined area provides information about the first time a specific AOI is seen. This can be beneficial for assessing what is being attended to in a visual scene (as opposed to what is not being attended to).

[0056] The embodiment of the present invention extracts eye movement information collected by the user of the virtual reality device through an eye tracker, determines the user's focus area in a fixed field of view, and aligns the visual center of the panoramic video frame with the focus area. When the user's eyeballs move, the panoramic rendering content is adaptively adjusted according to the eye movement trajectory, so as to avoid dizziness caused by the virtual reality solution under the user's natural movement.

[0057] Based on any of the above embodiments, the near-eye display of the fourth panoramic video frame includes: determining a third viewing angle range and a fourth viewing angle range according to the eye movement information.

[0058] Among them, the third viewing angle range is determined based on the left eye field of view of the user of the virtual reality device at the current eye position; the fourth viewing angle range is determined based on the right eye field of view of the user of the virtual reality device at the current eye position.

[0059] Specifically, the virtual reality device 130 uses the eye movement information to respectively calculate a third viewing angle range centered on the left eye's sight line at the current eyeball position, and a fourth viewing angle range centered on the right eye's sight line at the current eyeball position.

[0060] A third sub-video frame is captured from the third panoramic video frame using the third viewing angle range, and a fourth sub-video frame is captured from the third panoramic video frame using the fourth viewing angle range. The third sub-video frame and the fourth sub-video frame are alternately displayed in the left field and the right field of the virtual reality device.

[0061] Specifically, the virtual reality device 130 uses the third viewing angle range to capture a third sub-video frame with the left eye as the main visual center from the third panoramic video frame of 360°, and uses the fourth viewing angle range to capture a fourth sub-video frame with the right eye as the main visual center from the third panoramic video frame of 360°, and sends a stereo signal (left eye->right eye->left eye->right eye->in sequence and continuously alternately) to the screen of the near-eye display through the control IC, and simultaneously sends a synchronization signal to the virtual reality device 130 to synchronize the left and right eye images, so that the left eye sees the scene that the left eye should see (i.e., the third sub-video frame), and the right eye sees the scene that the right eye should see (i.e., the fourth sub-video frame).

[0062] The embodiment of the present invention determines the third viewing angle range and the fourth viewing angle range corresponding to the left and right eyes respectively according to the eye movement information, and uses the third viewing angle range and the fourth viewing angle range to intercept the third sub-video frame and the fourth sub-video frame from the third panoramic video frame, so that the left and right eye images are continuously and alternately displayed on the screen. Combined with the physiological characteristics of the visual persistence of the human eye, it can simulate a real stereoscopic 3D image.

[0063] Based on any of the above embodiments, the on-board processor performs the following steps when using the original image sequence to reconstruct the three-dimensional scene to obtain the first panoramic video frame: based on the field of view overlapping area between the first original image frame and the second original image frame in the original image sequence, obtaining the alignment pixel pair.

[0064] The first original image frame and the second original image frame are any two adjacent image frames in the original image sequence.

[0065] Specifically, the on-board processor 120 captures any image frame from the original image sequence as the first original image frame, and the next image frame adjacent to the image frame as the second original image frame, so that the on-board processor 120 extracts two groups of feature points from the image content, features, structure, relationship, texture and grayscale features of the first original image frame and the second original image frame, and then finds matching feature point pairs from the two groups of feature points by performing similarity measurement, and then obtains image space coordinate transformation parameters through the matching feature point pairs, and finally performs image registration through the coordinate transformation parameters to obtain corresponding registration pixel pairs.

[0066] The embodiment of the present invention does not specifically limit the properties of the feature points.

[0067] Exemplarily, feature points can be extracted by global analysis of image frames, and the feature points include but are not limited to Speeded Up Robust Feature (SURF), Scale Invariant Feature Transform (SIFT), Features from Accelerated Segment Test (FAST), Oriented FAST and Rotated BRIEF (ORB) and Harris corner points.

[0068] Exemplarily, feature points can be extracted from the overlapping area between two image frames by local analysis of image frames, and the feature points include but are not limited to Speeded Up Robust Feature (SURF), Scale Invariant Feature Transform (SIFT), Features from Accelerated Segment Test (FAST), Oriented FAST and Rotated BRIEF (ORB) and Harris corner points.

[0069] The first original image frame and the second original image frame are fused frame by frame by using the registered pixel pair until the second original image frame is the last original image frame in the original image sequence, thereby obtaining the first panoramic video frame.

[0070] Specifically, the on-board processor 120 uses the registered pixel pairs to perform a smooth transition, gradually transitioning from the first original image frame to the second original image frame in the overlapping area of ​​the field of view, that is, adding the pixel values ​​of the overlapping area of ​​the image according to a certain weight to synthesize a new image, and so on, using the above steps to fuse the images in the original image sequence frame by frame until the second original image frame is the last original image frame in the original image sequence, indicating that the fusion is completed to output the first panoramic video frame.

[0071] The embodiment of the present invention extracts the registration pixel pairs from the overlapping area of ​​the field of view between any two adjacent image frames in the original image sequence, and smoothly fuses the corresponding two adjacent image frames frame by frame according to the registration pixel pairs to obtain the first panoramic video frame representing the scene around the vehicle. By narrowing the range of matching feature point pairs, the accuracy and efficiency of image fusion can be improved, the fusion effect of images with a wide field of view or panoramic images can be optimized, and the delay of video splicing can be reduced.

[0072] Based on any of the above embodiments, obtaining the registration pixel pairs based on the field of view overlapping area between the first original image frame and the second original image frame in the original image sequence includes: determining the optimal registration line pixel position based on the first original image frame and the second original image frame.

[0073] Specifically, since the pan-tilt camera 110 realizes image acquisition at a constant frequency during the panoramic scanning process, in order to realize panoramic imaging of any airspace, there must be overlapping fields of view between adjacent frames. Within a reasonable range, no matter how large the pitch angle is, adjacent frames have intersection uniqueness in three-dimensional space. For any physical imaging scenic spot whose overlapping fields of view match the first original image frame and the second original image frame, due to the perspective deviation, the imaging effect of this point at different imaging positions of adjacent frames is different. However, the imaging effect of this point is the same at the intersection of adjacent frames.

[0074] In the world coordinate system XYZ, when the pitch angle of the pan-tilt camera 110 is α, assuming that the number of frames scanned by the pan-tilt camera 110 is N, the steering angle of the optical axis between adjacent frames is , so the second original image frame The azimuth is: The value that satisfies this condition is the first original image frame. The azimuth is and Since adjacent frames have unique spatial intersection, the spatial coordinates of the intersection satisfy , and , then the onboard processor 120 calculates the best registration line MN between adjacent frames in the image Pixel position , and in the image Pixel position It can be interpreted as: (1)

[0075] (2)

[0076] (3)

[0077] Among them, when hour, and There is a unique solution. At this point, the pixel position of the MN registration line can be searched in adjacent frames. Since the steering angle of the imaging system is constant, the pixel position of the registration line only needs to be solved once at the same pitch angle, which greatly reduces the computational complexity of full-frame stitching.

[0078] Based on the optimal registration line pixel position, the field of view overlap area between the first original image frame and the second original image frame is determined, so that the on-board processor can extract the registration pixel pairs from the first original image frame and the second original image frame according to the field of view overlap area.

[0079] Specifically, due to the non-orthogonality between the rotation axis and the optical axis during the circumferential scanning of the pan-tilt camera 110, there is an image rotation phenomenon when adjacent frames are located in a two-dimensional coordinate system, and the higher the elevation angle, the larger the overlapping area.

[0080] For example, the second original image frame After rotating by angle θ, the adjacent frames are translated so that and The first and last rows of the image overlap at points M2 and N2, and the overlapping area B2M2C1FN2E with the line between M2 and N2 as the axis of symmetry is a symmetrical hexagon. Therefore, the registration formula constructed by the on-board processor 120 for the overlapping area of ​​adjacent image frames is: (4)

[0081] (5)

[0082] Where: The first original image frame The pixel coordinates of The second original image frame The pixel coordinates of . and The best alignment line can be determined according to equations (1) to (3), that is, when hour, , .when hour, , .

[0083] The onboard processor 120 extracts feature points in the overlapping area of ​​the field of view from the first original image frame and the second original image frame according to the pixel position of the best registration line, and performs registration using the registration formula represented by equations (4) to (5) to obtain a registered pixel pair.

[0084] The embodiment of the present invention locks the overlapping area of ​​the field of view by determining the pixel position of the best registration line between any two adjacent image frames in the original image sequence, and performs registration after extracting feature points in the area, so as to obtain a pair of registration pixels with a high registration success rate, so as to smoothly fuse the corresponding two adjacent image frames frame by frame according to the registration pixel pairs, and obtain the first panoramic video frame representing the scene around the vehicle. The accuracy and efficiency of image fusion can be improved by narrowing the range of matching feature point pairs, optimizing the fusion effect for images with a wide field of view or panoramic images, and reducing the delay of video stitching.

[0085] On the basis of any of the above embodiments, the first original image frame and the second original image frame are fused frame by frame using the registered pixel pair until the second original image frame is the last original image frame in the original image sequence to obtain the first panoramic video frame, including: using the registered pixel pair to fuse the first original image frame and the second original image frame to obtain a first fused image frame.

[0086] Specifically, the onboard processor 120 uses an image fusion algorithm to merge and splice the overlapping areas of adjacent frames line by line to eliminate obvious splicing gaps. The boundary coordinates of the overlapping area and the registered pixel pairs can be calculated according to the above registration formula. is the fused image, which can be expressed as: (6)

[0087] (7)

[0088] in, is the weight gradient factor, is the left boundary coordinate of the overlapping area of ​​row y, The right boundary coordinate of the overlapping area in row y. is the number of overlapping pixels in the yth row, that is .

[0089] The first original image frame is updated to the first fused image frame, and the second original image frame is updated to the second original image frame and fused after the next image frame in the original image sequence, until the second original image frame is the last original image frame in the original image sequence, to obtain a second fused image frame.

[0090] Specifically, the on-board processor 120 can use the obtained first fused image frame as a new first original image frame, and use the next frame of the original second original image frame in the original image sequence as a new second original image frame, and repeat the above steps to perform image fusion until the last original image frame in the original image sequence is updated to a new second original image to achieve panoramic fusion and obtain a second fused image frame.

[0091] The second fused image frame is straightened based on the pixel position of the best registration line to determine the first panoramic video frame.

[0092] Specifically, the onboard processor 120 performs straightening processing on the spliced ​​second fused image frame based on the pixel position of the best registration line. Assume that the pixel coordinates of the best registration lines M1N1 and M2N2 are (M 1.x , M 1.y )、(N 1.x , N 1.y )、(M2.x , M 2.y )、(N 2.x , N 2.y ), let the coordinates after straightening be (K, 0), (K, H-1), (W+K-1, 0), (W+K-1, H-1), where K is a constant. According to the coordinate pairs before and after straightening, the panoramic image straightening is realized by using the transmission transformation method.

[0093] The embodiment of the present invention locks the overlapping area of ​​the field of view by determining the pixel position of the best registration line between any two adjacent image frames in the original image sequence, and performs registration after extracting feature points in the area, so as to obtain a pair of registration pixels with a high registration success rate, so as to smoothly fuse the corresponding two adjacent image frames frame by frame according to the registration pixel pairs, and obtain the first panoramic video frame representing the scene around the vehicle. The accuracy and efficiency of image fusion can be improved by narrowing the range of matching feature point pairs, optimizing the fusion effect for images with a wide field of view or panoramic images, and reducing the delay of video stitching.

[0094] Based on any of the above embodiments, the pan-tilt camera 110 includes a pan-tilt stabilizer and a camera module arranged on the pan-tilt stabilizer.

[0095] The gimbal stabilizer drives the camera module to rotate around a rotation axis so that the camera module can capture an original image sequence corresponding to the rotation axis.

[0096] The vehicle-mounted processor 120 is further used to fuse original image sequences corresponding to different rotation axes; Wherein, the rotation axis includes at least one of a pitch axis, a yaw axis and a roll axis.

[0097] Specifically, the PTZ camera 110 is composed of a PTZ stabilizer and a camera module, the optical axis of the camera module is perpendicular to the target surface of the PTZ stabilizer and intersects at the center of the target surface. The PTZ stabilizer can realize all-round uninterrupted imaging by rotating around the yaw axis, can realize imaging in a specified airspace by rotating around the pitch axis, and can realize imaging in a specified plane by rotating around the roll axis, so that the camera module forms a shooting mode of "azimuth scanning, pitch adjustable".

[0098] For example, the on-board processor 120 first fuses the original image sequence captured by the camera module around each rotation axis into a 360° circular panoramic image of the corresponding plane, and then stitches and fuses the circular panoramic images corresponding to different rotation axes to form a fused panoramic image of each pitch angle from top to bottom.

[0099] The embodiment of the present invention performs all-round image acquisition around one or more of the pitch axis, the yaw axis and the roll axis to stitch the two-dimensional plane pictures formed by each axis into a three-dimensional panoramic image, which can maximize the display of a more realistic scene and enhance the sense of immersion.

[0100] Figure 2 Schematic diagram of the structure of the vehicle provided by the present invention. Figure 2 As shown, based on any of the above embodiments, the vehicle provided in the embodiment of the present invention includes a vehicle body 210, and a vehicle-mounted panoramic monitoring system 220 loaded in the vehicle body 210 and linked with the virtual reality device as described in the above embodiments.

[0101] Specifically, the vehicle-mounted panoramic monitoring system 220 linked to the virtual reality device is installed on the vehicle body 210 to process the perceived information during the process of external perception, and project a panoramic video stream outside the vehicle in the corresponding direction to the corresponding user through the virtual reality device worn by the user inside the vehicle body 210, so as to obtain the real-time situation outside the vehicle.

[0102] Furthermore, each user inside the vehicle body 210 can express the behavioral intention of the actual task requirements by performing corresponding actions according to their actual task requirements. The virtual reality device they wear will then capture the action changes to parse the corresponding intentions and automatically adjust the visual center of the image displayed near the eye accordingly, so that multiple users can focus on their respective areas of concern at the same time.

[0103] The embodiment of the present invention senses the external scene of the vehicle through the pan-tilt camera, and the vehicle-mounted processor reconstructs the three-dimensional scene of the sensed original image sequence, and synchronizes the obtained first panoramic video frame in the form of a video stream to the virtual reality device worn by each user in the vehicle, and then the virtual reality device aligns the first panoramic video frame with the position of the user in the vehicle, and then displays the obtained second panoramic video frame near the eye. The panoramic video stream is always kept consistent with the user's position during the movement of the vehicle, greatly reducing the difference in human eye perception, and giving the viewer a sense of presence and immersion.

[0104] In addition, the above-mentioned logical instructions can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-mounted panoramic monitoring system linked with a virtual reality device, characterized in that: It includes a pan-tilt camera and an on-board processor arranged on a vehicle body, and a plurality of virtual reality devices communicatively connected to the on-board processor; The pan / tilt camera is used for panoramic scanning of the external scene of the vehicle body and transmitting the acquired original image sequence to the vehicle-mounted processor; The vehicle-mounted processor is used to synchronize the first panoramic video frame obtained by reconstructing the three-dimensional scene using the original image sequence to each of the virtual reality devices; The virtual reality device is used to adjust the first panoramic video frame to a second panoramic video frame corresponding to a fixed field of view of a user of the virtual reality device in the vehicle body, and to perform near-eye display of the second panoramic video frame; The fixed field of view is determined after aligning the first panoramic video frame with a polar coordinate system, and the polar coordinate system is constructed with the front of the vehicle as a direction of 0 degrees.

2. The vehicle-mounted panoramic monitoring system linked with a virtual reality device according to claim 1, characterized in that: An inertial navigation device is arranged on the virtual reality device, and the inertial navigation device is aligned with the polar coordinate system; The virtual reality device is further used to collect head posture information according to the inertial navigation device, adjust the second panoramic video frame to a third panoramic video frame according to the head posture information, and perform near-eye display of the third panoramic video frame.

3. According to the vehicle-mounted panoramic monitoring system linked with a virtual reality device in claim 2, the near-eye display of the third panoramic video frame comprises: Determine a first viewing angle range and a second viewing angle range according to the head posture information; Using the first viewing angle range to extract a first sub-video frame from the third panoramic video frame, using the second viewing angle range to extract a second sub-video frame from the third panoramic video frame, and alternately displaying the first sub-video frame and the second sub-video frame in the left image field and the right image field of the virtual reality device; The first viewing angle range is determined based on the left eye field of view of the user of the virtual reality device under the current head posture; the second viewing angle range is determined based on the right eye field of view of the user of the virtual reality device under the current head posture.

4. The vehicle-mounted panoramic monitoring system linked with a virtual reality device according to claim 1, characterized in that: An eye tracker is arranged in the virtual reality device; The virtual reality device is further used to collect eye movement information according to the eye tracker, adjust the second panoramic video frame to a fourth panoramic video frame according to the eye movement information, and perform near-eye display of the fourth panoramic video frame.

5. According to the vehicle-mounted panoramic monitoring system linked with a virtual reality device in claim 4, the near-eye display of the fourth panoramic video frame comprises: Determining a third viewing angle range and a fourth viewing angle range according to the eye movement information; Using the third viewing angle range to extract a third sub-video frame from the third panoramic video frame, using the fourth viewing angle range to extract a fourth sub-video frame from the third panoramic video frame, and alternately displaying the third sub-video frame and the fourth sub-video frame in the left image field and the right image field of the virtual reality device; Among them, the third viewing angle range is determined based on the left eye field of view of the user of the virtual reality device at the current eye position; the fourth viewing angle range is determined based on the right eye field of view of the user of the virtual reality device at the current eye position.

6. According to the vehicle-mounted panoramic monitoring system linked with a virtual reality device in claim 1, when the vehicle-mounted processor uses the original image sequence to reconstruct the three-dimensional scene to obtain the first panoramic video frame, the vehicle-mounted processor performs the following steps: Acquire a registration pixel pair based on a field of view overlap area between a first original image frame and a second original image frame in the original image sequence; Using the registered pixel pairs, the first original image frame and the second original image frame are fused frame by frame until the second original image frame is the last original image frame in the original image sequence, thereby obtaining the first panoramic video frame; in, The first original image frame and the second original image frame are any two adjacent image frames in the original image sequence.

7. According to the vehicle-mounted panoramic monitoring system linked to a virtual reality device in claim 6, the step of obtaining the registration pixel pair based on the overlapping area of ​​the field of view between the first original image frame and the second original image frame in the original image sequence comprises: Determining an optimal registration line pixel position based on the first original image frame and the second original image frame; Based on the optimal registration line pixel position, the field of view overlap area between the first original image frame and the second original image frame is determined, so that the on-board processor can extract the registration pixel pairs from the first original image frame and the second original image frame according to the field of view overlap area.

8. According to the vehicle-mounted panoramic monitoring system linked to a virtual reality device in claim 6, the step of using the registered pixel pairs to fuse the first original image frame and the second original image frame frame by frame until the second original image frame is the last original image frame in the original image sequence to obtain the first panoramic video frame comprises: fusing the first original image frame and the second original image frame using the registered pixel pairs to obtain a first fused image frame; Updating the first original image frame to the first fused image frame, and updating the second original image frame to the second original image frame to be fused after the next image frame in the original image sequence, until the second original image frame is the last original image frame in the original image sequence, to obtain a second fused image frame; The second fused image frame is straightened based on the pixel position of the best registration line to determine the first panoramic video frame.

9. According to the vehicle-mounted panoramic monitoring system linked with a virtual reality device as claimed in claim 1, the pan-tilt camera comprises a pan-tilt stabilizer and a camera module arranged on the pan-tilt stabilizer; The gimbal stabilizer drives the camera module to rotate around the rotation axis, so that the camera module can collect an original image sequence corresponding to the rotation axis; The vehicle-mounted processor is further used to fuse original image sequences corresponding to different rotation axes; in, The rotation axis includes at least one of a pitch axis, a yaw axis and a roll axis.

10. A vehicle, characterized in that: It comprises a vehicle body, and a vehicle-mounted panoramic monitoring system linked with a virtual reality device as claimed in any one of claims 1 to 9, which is loaded in the vehicle body.

Images