Blind area display method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a blind area display method and device, electronic equipment and a readable storage medium, and relates to the technical field of display. The method comprises the steps of determining a view blind area corresponding to a target person based on augmented reality equipment worn by the target person in a target vehicle; acquiring a real-scene image corresponding to the view blind area; predicting a target image of a specified timestamp based on the live-action image, the specified timestamp being a timestamp after an initial timestamp, and the initial timestamp being a timestamp for obtaining the live-action image; and displaying the target image through the augmented reality device. The target image is obtained by predicting the real scene image of the view blind area, and then the target image is displayed in the augmented reality device, so that the position difference between the target image displayed in the augmented reality device and the real scene corresponding to the view blind area is reduced.

Description

Technical Field

[0001] The present application relates to the field of display technology, and more specifically, to a blind area display method, device, electronic device and readable storage medium. Background Art

[0002] At present, there are objects blocking the driver's view, which will cause the driver to have a blind spot. Although the image of the blind spot can be collected by the sensor equipment and displayed to the driver, the current method of displaying the image of the blind spot has a delay in display, which leads to a difference between the image displayed to the driver and the real scene corresponding to the blind spot. Summary of the invention

[0003] The present application proposes a blind spot display method, device, electronic device and readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides a blind spot display method, the method comprising: determining a blind spot corresponding to a target person in a target vehicle based on an augmented reality device worn by the target person; acquiring a real-life image corresponding to the blind spot; predicting a target image at a specified timestamp based on the real-life image, the specified timestamp being a timestamp after an initial timestamp, the initial timestamp being a timestamp when the real-life image is acquired; and displaying the target image through the augmented reality device.

[0005] Optionally, for a possible implementation, predicting a target image of a specified timestamp based on the real-scene image includes: acquiring target three-dimensional optical flow information of each pixel in the real-scene image based on the real-scene image and pre-acquired specified extrinsic parameters, wherein the specified extrinsic parameters are extrinsic parameters corresponding to an acquisition device used to acquire the real-scene image, and the target three-dimensional optical flow information includes first position information and first speed information of each pixel in a world coordinate system; generating a predicted target image based on the target three-dimensional optical flow information, an initial timestamp and the specified timestamp.

[0006] Optionally, for a possible implementation, the acquiring of target three-dimensional optical flow information of each pixel in the real-scene image based on the real-scene image and pre-acquired designated external parameters includes: acquiring the movement speed of the target vehicle; determining the two-dimensional optical flow information of each pixel in the real-scene image based on the real-scene image and the designated external parameters, wherein the two-dimensional optical flow information includes the second position information and second speed information of each pixel in the image coordinate system, and the image coordinate system is a coordinate system established based on the real-scene image; determining the third position information and third speed information of each pixel in the three-dimensional vector field in the world coordinate system based on the designated external parameters, the movement speed and the two-dimensional optical flow information, wherein the pixels included in the three-dimensional vector field correspond to the pixels included in the real-scene image; determining the target three-dimensional optical flow information based on the third position information and the third speed information.

[0007] Optionally, for a possible implementation, determining the two-dimensional optical flow information of each pixel in the real-scene image based on the real-scene image and the specified external parameters includes: obtaining fourth position information of the real scene corresponding to the real-scene image and fourth speed information of the real scene relative to the target vehicle, wherein the fourth position information and the fourth speed information are both in the world coordinate system; converting the fourth speed information into fifth speed information in the acquisition device coordinate system based on the specified external parameters, and converting the fourth position information into fifth position information in the acquisition device coordinate system based on the specified external parameters; converting the fifth speed information into second speed information in the image coordinate system based on a projection function, and converting the fifth position information into second position information in the image coordinate system based on the projection function; and using the second speed information and the second position information as the two-dimensional optical flow information.

[0008] Optionally, for a possible implementation, the real scene includes a first real scene and a second real scene, and obtaining the fourth position information of the real scene corresponding to the real scene image and the fourth speed information of the real scene relative to the target vehicle includes: determining the position information of the first real scene corresponding to the first image area and the position information of the second real scene corresponding to the second image area in the world coordinate system, respectively, wherein the first image area is the area including the target object in the real scene image, and the second image area is the area not including the target object in the real scene image; using the position information of the first real scene and the position information of the second real scene as the fourth position information; obtaining the speed information of the first real scene corresponding to the first image area captured by the target vehicle; using the movement speed of the target vehicle as the speed information of the second real scene corresponding to the second image area, and using the speed information of the first real scene and the speed information of the second real scene as the fourth speed information.

[0009] Optionally, for a possible implementation, determining the target three-dimensional optical flow information based on the third position information and the third speed information includes: using the third position information and the third speed information as initial three-dimensional optical flow information; and performing continuous frame smoothing on the initial three-dimensional optical flow information to obtain the target three-dimensional optical flow information.

[0010] Optionally, for a possible implementation, generating a predicted target image based on the target three-dimensional optical flow information, the initial timestamp and the specified timestamp includes: taking the difference between the specified timestamp and the initial timestamp as a difference timestamp; taking the target three-dimensional optical flow information and the difference timestamp as input quantities, inputting them into a pre-trained deep learning model, and obtaining the output quantity of the deep learning model as the target image.

[0011] Optionally, for a possible implementation, before predicting the target image with a specified timestamp based on the real-scene image, it also includes: predicting a historical real-scene image that has been collected in advance, and displaying the predicted historical real-scene image in the augmented reality device; determining a historical timestamp when the historical real-scene image is collected; determining an intermediate timestamp for displaying the predicted real-scene image in the augmented reality device; and using the difference between the intermediate timestamp and the historical timestamp as the specified timestamp.

[0012] Optionally, for a possible implementation, the determining of the blind spot of vision corresponding to the target person based on the augmented reality device worn by the target person in the target vehicle includes: determining the posture information of the target person's eyes based on the augmented reality device; and calculating the area corresponding to the occluded object in the target vehicle through a perspective projection algorithm based on the posture information as the blind spot of vision.

[0013] Optionally, for a possible implementation, displaying the target image through the augmented reality device includes: performing image processing on the target image to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising; and displaying the target image after image processing through the augmented reality device.

[0014] In the second aspect, the embodiment of the present application also provides a blind spot display device, the device comprising: a determination unit, an acquisition unit, a prediction unit and a display unit. The determination unit is used to determine the blind spot corresponding to the target person in the target vehicle based on the augmented reality device worn by the target person; the acquisition unit is used to acquire the real scene image corresponding to the blind spot; the prediction unit is used to predict the target image of a specified timestamp based on the real scene image, the specified timestamp is the timestamp after the initial timestamp, and the initial timestamp is the timestamp when the real scene image is acquired; the display unit is used to display the target image through the augmented reality device.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method described in the first aspect above.

[0017] The blind spot display method, device, electronic device and readable storage medium provided in the embodiment of the present application, the method first determines the blind spot corresponding to the target person in the target vehicle based on the augmented reality device worn by the target person; then obtains the real scene image corresponding to the blind spot; and then predicts the target image of the specified timestamp based on the real scene image; thereby displaying the target image through the augmented reality device. If the image of the blind spot collected is directly displayed to the target person, it takes a certain amount of time from collecting the image of the blind spot to displaying the image to the target person, and the real scene corresponding to the image of the blind spot is generally moving relative to the vehicle, which may cause display delays, so that there is a position difference between the image displayed to the target person and the real scene corresponding to the currently blocked blind spot. In the blind spot display method provided in the present application, the target image of the specified timestamp can be predicted based on the real scene image, and the specified timestamp is a timestamp after the initial timestamp, and the initial timestamp is the timestamp when the real scene image is obtained. That is to say, the target image finally displayed to the augmented reality device is an image obtained after predictive processing of the real scene image in the blind spot, thereby reducing the position difference between the target image displayed in the augmented reality device and the real scene corresponding to the blind spot.

[0018] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 An application scenario diagram of the blind spot display method provided by an embodiment of the present application is shown;

[0021] Figure 2 A method flow chart of a blind spot display method provided by an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram showing a blind area of ​​vision provided by an embodiment of the present application is shown;

[0023] Figure 4 A method flow chart of a blind area display method provided by another embodiment of the present application is shown;

[0024] Figure 5 A schematic diagram of a real scene image provided by an embodiment of the present application is shown;

[0025] Figure 6 A schematic diagram showing target three-dimensional optical flow information provided by an embodiment of the present application is shown;

[0026] Figure 7 A schematic diagram of a target image provided by an embodiment of the present application is shown;

[0027] Figure 8 A method flow chart of a blind area display method provided by another embodiment of the present application is shown;

[0028] Fig. 9 A structural block diagram of a blind spot display device provided in an embodiment of the present application is shown;

[0029] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0030] Fig.11 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] At present, there are objects blocking the driver's view, which will cause the driver to have a blind spot. Although the image of the blind spot can be collected by the sensor equipment and displayed to the driver, the current method of displaying the image of the blind spot has a delay in display, which leads to a difference between the image displayed to the driver and the real scene corresponding to the blind spot. How to reduce the difference between the image displayed to the driver and the real scene corresponding to the blind spot is an urgent problem to be solved.

[0034] Currently, real-life images of blind spots can be collected and then displayed to the driver.

[0035] However, the inventors discovered during their research that it takes a certain amount of time from capturing the real-scene image to displaying it to the driver, and the real scene in the blind spot is generally moving relative to the vehicle, which can cause a position difference between the real-scene image displayed to the driver and the real scene corresponding to the blind spot, that is, the accuracy of the displayed image content is low.

[0036] Therefore, in order to solve or partially solve the above problems, the embodiments of the present application provide a blind area display method, device, electronic device and readable storage medium.

[0037] See also Figure 1 , Figure 1 An application scenario diagram of the blind spot display method provided in an embodiment of the present application, that is, a blind spot display scenario 100 is shown. Figure 1, a blind spot display scene 100 is shown, and the blind spot display scene 100 includes a target vehicle 110 and a pedestrian 120, and the pedestrian 120 is located at the zebra crossing 190 and passes normally. There is a blind spot 180 for the target person in the target vehicle 110 to be observed through the target vehicle, wherein the target person may be the driver of the target vehicle 110. At this time, the pedestrian 120 is located in the blind spot 180, and the driver driving the target vehicle 110 has the problem of not being able to observe the pedestrian 120, which may cause a safety accident. Therefore, the blind spot display method provided in the embodiment of the present application can be used to display the real scene corresponding to the blind spot 180 to the driver, thereby helping to improve the driver's cognitive ability of the environment surrounding the target vehicle 110, and can effectively reduce traffic accidents caused by the blind spot 180, and improve driving safety. Among them, for a detailed introduction to the blind spot display method, please refer to the subsequent method embodiments.

[0038] See also Figure 2 , Figure 2 A method flow chart of a blind area display method provided by an embodiment of the present application is shown. The blind area display method can be applied to Figure 1 In the blind spot display scenario shown in , the method may specifically include steps S110 to S140.

[0039] Step S110: Based on the augmented reality device worn by the target person in the target vehicle, determine the blind spot of the field of vision corresponding to the target person.

[0040] The target vehicle is a car, which can be a traditional oil car or a new energy car, which is not specifically limited in the embodiments of the present application. The target person in the target vehicle can be the driver driving the target vehicle, or a passenger riding in the target vehicle, such as a passenger sitting in the co-pilot seat, or a passenger sitting in the back seat, etc. It should be noted that the subsequent introduction takes the target person as the driver driving the target vehicle as an example for specific description.

[0041] The eyes of the target person can be characterized by posture information. The posture information can specifically include position information and posture information. The position information can be characterized by the position of the eyes of the target person in the vehicle coordinate system, and the posture information can be characterized by the roll angle, pitch angle and heading angle of the target person. The vehicle coordinate system is a coordinate system established based on the target vehicle.

[0042] The target person's eyes will form a blind spot when they pass through the obstructing objects in the target vehicle. For example, see Figure 3 , Figure 3 A schematic diagram showing a blind area of ​​vision in an embodiment of the present application is shown. Figure 3Target vehicle 110 and target person's eyes 310 are shown in FIG. There is an obstructing object 111 in target vehicle 110, so that the human eye 310 and the obstructing object 111 form a visual blind spot 180. Figure 3 The blocking object 111 shown in FIG. 1 is the A-pillar on the left side of the target vehicle.

[0043] It is understandable that the position information of the target person's eyes in the vehicle coordinate system will change. The change in the position information of the eyes will cause the blind area of ​​vision to change. Therefore, it is necessary to determine the blind area of ​​vision corresponding to the target person. It should be noted that the blind area of ​​vision corresponding to the target person is the blind area of ​​vision corresponding to the eyes of the target person.

[0044] As the technology related to augmented reality (AR) devices develops, the weight, battery life, performance, etc. of augmented reality devices are becoming more and more perfect. The target person wearing the augmented reality device can obtain the position information of the target person's eyes, and the augmented reality device can also display image content for the target person. Among them, the augmented reality device can be augmented reality glasses, head-mounted augmented reality displays, etc. Therefore, in some embodiments, the target person in the target vehicle can wear an augmented reality device, so that the position information of the target person's eyes can be obtained through the augmented reality device, and then the blind spot of the field of vision can be determined in combination with the object in the target vehicle. Exemplarily, the augmented reality device can be configured with a camera, so that the position information of the target person's glasses can be determined by the camera configured by the augmented reality device. For detailed introduction, please refer to the subsequent embodiments.

[0045] Step S120: Acquire a real scene image corresponding to the blind area of ​​vision.

[0046] It can be seen from the introduction of the above steps that it is not easy for the target person in the target vehicle to directly observe the real scene in the blind spot in the target vehicle, which may lead to the risk of traffic accidents. In order to display the image corresponding to the blind spot through the augmented reality device later, the real scene image corresponding to the blind spot can be obtained first, and then the prediction processing and other operations of the real scene image can be performed.

[0047] For some implementations, an initial image around the target vehicle may be first obtained, and then an image corresponding to the blind spot may be cropped from the initial image as the real scene image corresponding to the blind spot. For example, a camera may be provided on the outside of the target vehicle, so that the initial image around the target vehicle may be acquired by the camera provided on the outside of the target vehicle. For example, please continue to refer to Figure 3 , Figure 3A camera 112 is disposed on the outer side of the vehicle body of the target vehicle 110 , so that the initial image around the target vehicle can be collected by the camera 112 .

[0048] It should be noted that the number of cameras provided on the vehicle body may be one or more. In some embodiments, the images captured by the multiple cameras may be fused to obtain an initial image with a wider field of view. It is understood that the initial image captured by the camera provided on the vehicle body should at least include a real-life image corresponding to the blind spot of the field of view.

[0049] Step S130: predicting a target image of a specified timestamp based on the real scene image, where the specified timestamp is a timestamp after an initial timestamp, and the initial timestamp is a timestamp when the real scene image is acquired.

[0050] After obtaining the real-scene image corresponding to the blind spot, it takes a certain amount of time to display the acquired real-scene image corresponding to the blind spot in the augmented reality device. Therefore, if the real-scene image is displayed directly through the augmented reality device, the real-scene image displayed to the driver will have a position difference with the real scene corresponding to the blind spot, that is, the accuracy of the displayed image content is low.

[0051] Therefore, in the embodiments provided in the present application, the image corresponding to the timestamp after the timestamp of the real scene image is acquired can be predicted. Specifically, the prediction can be made based on the acquired real scene image to obtain the target image of the specified timestamp, and then the target image can be displayed later. Among them, the initial timestamp is the timestamp when the real scene image is acquired, and the specified timestamp is the timestamp after the initial timestamp. Therefore, what is displayed later is not the acquired real scene image, but the target image, and the specified timestamp corresponding to the target image is after the initial timestamp. Therefore, the position difference between the target image displayed to the driver and the real scene corresponding to the blind spot of the field of vision can be reduced, thereby improving the accuracy of the displayed image content. That is to say, delayed dynamic compensation of the real scene image is realized.

[0052] In some embodiments, the historical real-scene image acquired in advance can be predicted and directly displayed in the augmented reality device, thereby determining an intermediate timestamp for displaying the predicted historical real-scene image in the augmented reality device; and then using the difference between the intermediate timestamp and the initial timestamp as the designated timestamp. For a detailed description, please refer to the subsequent embodiments.

[0053] In some embodiments, after acquiring the real scene image, the target three-dimensional optical flow information of each pixel in the real scene image can be acquired, wherein the target three-dimensional optical flow information includes the first position information and the first speed information of each pixel in the world coordinate system. In other words, each pixel in the real scene image has a motion speed, so that at a specified timestamp a period of time after the initial timestamp, the position of each pixel can be predicted based on the target three-dimensional optical flow information, and each predicted pixel can constitute the target image. It should be noted that the first speed information is the speed information relative to the target vehicle. For a detailed description of the predicted target image, please refer to the subsequent embodiments.

[0054] Step S140: Displaying the target image through the augmented reality device.

[0055] Thus, the target image obtained in the above steps is an image predicted based on the specified timestamp on the basis of the real scene image. Displaying the target image by the augmented reality device can reduce the position difference between the displayed target image and the real scene corresponding to the blind spot of the visual field, thereby improving the accuracy of the displayed image content.

[0056] In some embodiments, the target person can obtain a first image through the augmented reality device. The first image can be an image of the real world that the target person directly sees through the augmented reality device. It is understandable that there is a blind spot in the first image, that is, there is an obstructed area. Then, the acquired target image can be spliced ​​to the area corresponding to the blind spot in the first image for display, that is, the real world directly seen by the target person is spliced ​​with the acquired target image and then displayed.

[0057] For example, in the case where the obstructing objects in the target vehicle are the A-pillars on both sides, the acquired target image is spliced ​​to the area corresponding to the blind spot in the first image for display, so that the target person of the target vehicle can observe the image in the blind spot determined based on the A-pillars on both sides, thereby better understanding the environment surrounding the vehicle.

[0058] In other embodiments, the target image may be directly displayed in a designated area of ​​the first image, and the transparency of the displayed target image may be adjusted, for example, to 50% transparency, so that the target image is displayed superimposed on the first image, and the first image below the target image will not be blocked by the target image. The designated area may be a pre-set area, such as the upper left corner area of ​​the first screen, the lower left corner area of ​​the first screen, etc.; it may also be adjusted and determined by the target person based on the pre-set area, and the embodiments of the present application do not specifically limit this.

[0059] The blind spot display method provided in the embodiment of the present application first determines the blind spot corresponding to the target person in the target vehicle based on the augmented reality device worn by the target person; then obtains the real scene image corresponding to the blind spot; and then predicts the target image of the specified timestamp based on the real scene image; thereby displaying the target image through the augmented reality device. If the acquired image of the blind spot is directly displayed to the driver, it takes a certain amount of time from acquiring the image of the blind spot to displaying the image to the driver, and the real scene corresponding to the image of the blind spot is generally moving relative to the vehicle, which may cause display delays, so that there is a position difference between the image displayed to the driver and the real scene corresponding to the currently blocked blind spot. In the blind spot display method provided in the present application, the target image of the specified timestamp can be predicted based on the real scene image, and the specified timestamp is a timestamp after the initial timestamp, and the initial timestamp is the timestamp when the real scene image is acquired. That is to say, the embodiments of the present application can acquire and predictively process real-scene images in real time, and the target image finally displayed to the augmented reality device is an image obtained after predictive processing of the real-scene image in the blind spot of the field of view, thereby reducing the position difference between the target image displayed in the augmented reality device and the real scene corresponding to the blind spot of the field of view.

[0060] See also Figure 4 , Figure 4 A method flow chart of a blind area display method provided by an embodiment of the present application is shown. The blind area display method can be applied to Figure 1 In the blind spot display scenario shown in , the method may specifically include steps S210 to S250.

[0061] Step S210: Based on the augmented reality device worn by the target person in the target vehicle, determine the blind spot of the field of vision corresponding to the target person.

[0062] Step S220: Acquire a real scene image corresponding to the blind spot of the visual field.

[0063] Among them, step S210 and step S220 have been introduced in detail in the above embodiments and will not be repeated here.

[0064] Step S230: Based on the real scene image and the pre-acquired specified external parameters, the target three-dimensional optical flow information of each pixel point in the real scene image is obtained, wherein the specified external parameters are external parameters corresponding to the acquisition device used to obtain the real scene image, and the target three-dimensional optical flow information includes the first position information and the first speed information of each pixel point in the world coordinate system.

[0065] In some embodiments, after acquiring the real scene image, the target three-dimensional optical flow information of each pixel in the real scene image can be acquired, wherein the target three-dimensional optical flow information includes the first position information and the first speed information of each pixel in the world coordinate system. It can be understood that the acquired real scene image is a two-dimensional image, and the target three-dimensional optical flow information is the first position information and the first speed information of each pixel in the world coordinate system, therefore, it is necessary to perform coordinate transformation on each pixel in the two-dimensional real scene image to obtain the target three-dimensional optical flow information.

[0066] Each pixel point in the two-dimensional real-scene image can be converted into the world coordinate system by specifying external parameters, thereby obtaining the target three-dimensional optical flow information. Among them, the specified external parameters are the external parameters corresponding to the acquisition device used to obtain the real-scene image. From the above introduction, it can be known that the acquisition device can be a camera or a camera, and the camera or the camera is set on the outside of the target vehicle body. Among them, the specified external parameters are the external parameters of the acquisition device coordinate system relative to the target vehicle. Exemplarily, the specified external parameters may include parameters such as rotation angle, translation vector and rotation matrix. It can be understood that the acquisition device coordinate system is a coordinate system established based on the acquisition device. If the acquisition device is a camera

[0067] Specifically, for some implementations, step S230 may also include steps S231 to S234.

[0068] Step S231: Acquire the moving speed of the target vehicle.

[0069] There are many methods for obtaining the moving speed of the target vehicle, for example, the moving speed can be obtained through an inertial measurement unit (IMU) on the target vehicle. For another example, the moving speed can also be obtained through a global positioning system (GPS) on the target vehicle, which is not specifically limited in the embodiments of the present application.

[0070] Step S232: Based on the real scene image and the specified external parameters, determine the two-dimensional optical flow information of each pixel point in the real scene image, wherein the two-dimensional optical flow information includes the second position information and second speed information of each pixel point in the image coordinate system, and the image coordinate system is a coordinate system established based on the real scene image.

[0071] In order to obtain the target three-dimensional optical flow information, the two-dimensional optical flow information of each pixel in the real scene image can be first obtained, wherein the two-dimensional optical flow information includes the second position information and the second speed information of each pixel in the image coordinate system, and the image coordinate system is a coordinate system established based on the real scene image.

[0072] It is understandable that each pixel point in the acquired real-scene image may have second speed information and second position information in the image coordinate system. However, the real scene corresponding to the real-scene image can be acquired through the target vehicle, and the speed information and position information of each pixel point of the real scene are all information in the world coordinate system. Therefore, it is also necessary to convert the acquired speed information in the world coordinate system into the second speed information in the image coordinate system, and convert the acquired position information in the world coordinate system into the second position information in the image coordinate system, so as to obtain the second optical flow information. It should be noted that the second speed information is the speed information relative to the target vehicle.

[0073] Specifically, in some implementations, step S232 may also include steps S2321 to S2324.

[0074] Step S2321: Acquire fourth position information of the real scene corresponding to the real scene image and fourth speed information of the real scene relative to the target vehicle, wherein the fourth position information and the fourth speed information are both in the world coordinate system.

[0075] For some implementations, the fourth position information of the real scene corresponding to the real scene image and the fourth speed information of the real scene relative to the target vehicle can be obtained through the detection module carried by the target vehicle, wherein the fourth position information and the fourth speed information are both in the world coordinate system.

[0076] It should be noted that the real scene corresponding to the real scene image can be understood as the specific object included in the real scene image. For example, if the real scene image is an image of a pedestrian, the real scene corresponding to the real scene image is the pedestrian; if the real scene image is an image of a vehicle, the real scene corresponding to the real scene image is the vehicle.

[0077] Exemplarily, the detection module carried by the target vehicle may include a laser sensor and a filter, so that the laser sensor is combined with semantic segmentation, and the data collected by the laser sensor is filtered by the filter to obtain the fourth position information and the fourth speed information. In some embodiments, the filter may be a Kalman filter.

[0078] It is understandable that there may be areas in the real-scene image that include target objects moving at a faster speed, and there may also be areas that do not include target objects moving at a faster speed. Therefore, the position information and speed information can be determined separately for different areas to improve the accuracy of the speed information of the acquired real-scene image.

[0079] Optionally, for some other implementations, step S2321 may also include steps S2325 to S2328.

[0080] Step S2325: Determine respectively in the world coordinate system the position information of the first real scene corresponding to the first image area and the position information of the second real scene corresponding to the second image area, wherein the first image area is the area including the target object in the real scene image, and the second image area is the area not including the target object in the real scene image.

[0081] Step S2326: taking the position information of the first real scene and the position information of the second real scene as the fourth position information.

[0082] Step S2327: Obtain speed information of a first real scene corresponding to a first image area captured by the target vehicle.

[0083] Coordinate S2328: The moving speed of the target vehicle is used as the speed information of the second real scene corresponding to the second image area, and the speed information of the first real scene and the speed information of the second real scene are used as the fourth speed information.

[0084] Optionally, the real scene may include a first real scene and a second real scene. The first real scene is the real scene corresponding to the first image area, and the second real scene is the real scene corresponding to the second image area. The first image area is the area including the target object in the real scene image, and the second image area is the area not including the target object in the real scene image. That is, the first real scene is an object moving faster in the world coordinate system, and the second real scene is an object that is stationary or almost stationary in the world coordinate system.

[0085] Exemplarily, if the real scene corresponding to the real scene image includes a traffic light and pedestrians, the first real scene may be the pedestrian, and the second real scene may be the traffic light. The first image area may be the area of ​​the real scene image including the pedestrian, and the second image area may be the area of ​​the real scene image including the traffic light.

[0086] That is, for some embodiments, the first image region and the second image region in the real scene image may be determined first. Exemplarily, the real scene image may be segmented by a bounding box, the region within the bounding box is the first image region, and the region outside the bounding box is the second image region.

[0087] Furthermore, the position information of the first real scene corresponding to the first image area and the position information of the second real scene corresponding to the second image area can be determined respectively in the world coordinate system, and then the position information of the first real scene and the position information of the second real scene are used as the fourth position information.

[0088] From the above analysis, it can be seen that the first real scene and the second real scene have different speed information, so the speed information of the first real scene corresponding to the first image area can be collected by the target vehicle. Similar to the above introduction, the speed information of the first real scene can be obtained by the detection module carried by the target vehicle.

[0089] Furthermore, since the second image area is an area that does not include the target object, the moving speed of the target vehicle can be used as the speed information of the second real scene. It should be noted that since the second real scene is stationary or almost stationary in the world coordinate system, the speed information of the second real scene relative to the target vehicle can be approximately regarded as the moving speed of the target vehicle. At this time, the speed information of the first real scene and the speed information of the second real scene can be used as the fourth speed information.

[0090] In this embodiment, the position information and the speed information may be determined for different areas respectively, so as to improve the accuracy of the speed information of the acquired real scene image.

[0091] Step S2322: converting the fourth speed information into fifth speed information in the acquisition device coordinate system based on the specified external parameter, and converting the fourth position information into fifth position information in the acquisition device coordinate system based on the specified external parameter.

[0092] Step S2323: converting the fifth speed information into second speed information in the image coordinate system based on a projection function, and converting the fifth position information into second position information in the image coordinate system based on a projection function.

[0093] Step S2324: using the second speed information and the second position information as the two-dimensional optical flow information.

[0094] The fourth speed information and the fourth position information obtained through the above steps are both in the world coordinate system. Therefore, in order to obtain the two-dimensional light flow information subsequently, the fourth speed information can be converted into the fifth speed information in the acquisition device coordinate system based on the specified external parameter, and the fourth position information can be converted into the fifth position information in the acquisition device coordinate system based on the specified external parameter.

[0095] Exemplarily, the fourth speed information can be converted into fifth speed information in the acquisition device coordinate system by reprojection conversion coordinates in combination with the specified external parameter, and the fourth position information can be converted into fifth position information in the acquisition device coordinate system based on the specified external parameter. For example, in the case where the real scene includes the first real scene and the second real scene, the first image area is characterized by the area within the bounding box. Thus, the bounding box can set multiple corner points, such as 8 corner points, and each corner point is converted to the acquisition device coordinate system by rotation and translation.

[0096] Further, the fifth speed information may be converted into second speed information in the image coordinate system based on a projection function, and the fifth position information may be converted into second position information in the image coordinate system based on a projection function.

[0097] Thus, the second speed information and the second position information can be used as the two-dimensional optical flow information. That is, the two-dimensional optical flow information of each pixel in the real scene image in the image coordinate system can be obtained. In some embodiments, the two-dimensional optical flow information of each pixel can be represented by a two-dimensional rectangle and the two-dimensional speed corresponding to the two-dimensional rectangle. The two-dimensional optical flow information of each pixel can also be called dense optical flow (Dense Optical Flow).

[0098] Each pixel in the real scene image can also be represented by a two-dimensional optical flow vector, specifically by the horizontal displacement and vertical displacement of each pixel in a unit time interval. It should be noted that the horizontal direction and the vertical direction are both determined based on the image coordinate system.

[0099] Step S233: Based on the specified external parameters, motion speed and two-dimensional optical flow information, determine the third position information and third speed information of each pixel point in the three-dimensional vector field in the world coordinate system, wherein the pixel points included in the three-dimensional vector field correspond to the pixel points included in the real scene image.

[0100] Furthermore, after the two-dimensional optical flow information is obtained, the third position information and the third speed information of each pixel point in the three-dimensional vector field in the world coordinate system may be determined.

[0101] Specifically, the two-dimensional motion of each pixel in the real-scene image can be solved by the two-dimensional optical flow information. Further combined with the specified external parameters and the movement speed of the target vehicle, the third position information and third speed information of each pixel in the real-scene image in the world coordinate system can be obtained. In some embodiments, the real-scene image in the world coordinate system can be represented by a three-dimensional vector field, so that the pixels included in the three-dimensional vector field correspond to the pixels included in the real-scene image, that is, the number of pixels included in the three-dimensional vector field is the same as the number of pixels included in the real-scene image.

[0102] In some implementations, the third position information and the third speed information may be calculated using the aforementioned two-dimensional optical flow vector.

[0103] Specifically, the third position information can be represented by the following formula (1).

[0104]

[0105] Among them, P in formula (1) w is used to represent the third position information, [u, v] is used to represent the second position information of the pixel point in the real scene image in the image coordinate system, Π is used to represent the conversion function from the image coordinate system to the world coordinate system, T wc Used to characterize the position and rotation of the acquisition device in the world coordinate system.

[0106] By taking the derivative of each term in equation (1), we can obtain equation (2).

[0107]

[0108] in, It is used to characterize the second speed information, and the other parts in formula (2) are all derivative functions of known functions, so that the third position information can be calculated.

[0109] Furthermore, the third velocity information of each pixel in the three-dimensional vector field can be represented by (vx, vy, vz).

[0110] Step S234: Determine the target three-dimensional optical flow information based on the third position information and the third speed information.

[0111] Furthermore, the acquired third position information and third speed information may be directly used as target three-dimensional optical flow information.

[0112] Optionally, in some implementations, the third speed information of each pixel point may be corrected, so that the third position information and the corrected third speed information are used as the target three-dimensional optical flow information.

[0113] Among them, the speed information of the faster pixel points can be collected by the target vehicle, for example, the speed information of the first real scene corresponding to the first image area can be collected by the target vehicle. Similarly, the speed information of the first real scene can be obtained by the detection module carried by the target vehicle. Then, the third speed information is corrected by the collected speed information of the first real scene. For example, the correction can be made by the following formula (3).

[0114]

[0115] in, is the corrected third speed information; f is a correction function, and illustratively, f may be a weighted average function, which is not specifically limited in the present embodiment; V sensor That is, the speed information of the first real scene collected; That is the third speed information, thereby obtaining more accurate target three-dimensional optical flow information.

[0116] Optionally, since the acquired third position information and third speed information are instantaneous information of each pixel in the three-dimensional vector field, if the acquired third position information and third speed information are directly used as the target three-dimensional optical flow information, the target three-dimensional optical flow information may be jittered or noisy, etc., which reduces the accuracy of the acquired target three-dimensional optical flow information, thereby resulting in poor effect of the target image obtained by subsequent prediction. Therefore, in some embodiments, the target three-dimensional optical flow information can also be obtained by continuous frame smoothing.

[0117] Specifically, when executing step S234, step S2341 and step S2342 may also be included.

[0118] Step S2341: using the third position information and the third speed information as initial three-dimensional optical flow information.

[0119] Step S2342: performing continuous frame smoothing processing on the initial three-dimensional optical flow information to obtain the target three-dimensional optical flow information.

[0120] First, the third position information and the third speed information are used as initial three-dimensional optical flow information. Then, continuous frame smoothing processing can be performed on the initial three-dimensional optical flow information to obtain the target three-dimensional optical flow information.

[0121] In some embodiments, a deep learning optimization model can be pre-trained based on deep learning, so that the initial three-dimensional optical flow information can be directly smoothed by the deep learning optimization model to obtain the target three-dimensional optical flow information. Specifically, the initial three-dimensional optical flow information can be used as the input of the deep learning optimization model, and the output of the deep learning optimization model is the target three-dimensional optical flow information. Among them, the deep learning optimization model can use a deep learning convolutional neural network to optimize the initial three-dimensional optical flow information.

[0122] The implementation manner of the present application obtains target three-dimensional optical flow information through continuous frame smoothing, thereby improving the accuracy of the obtained target three-dimensional optical flow information, thereby improving the effect of the target image obtained by subsequent prediction to a certain extent.

[0123] Step S240: Generate a predicted target image based on the target three-dimensional optical flow information, the initial timestamp and the designated timestamp.

[0124] From the above introduction, it can be known that the target three-dimensional optical flow information is the three-dimensional optical flow information of the real-scene image corresponding to the initial timestamp. Therefore, the timestamp after the initial timestamp can be predicted based on the target three-dimensional optical flow information, that is, the position information of each pixel point at the specified timestamp in the image coordinate system, and then the target image can be obtained.

[0125] For example, the difference timestamp can be obtained by specifying the difference between the timestamp and the initial timestamp, and the position information of each pixel in the acquisition device coordinate system of the specified timestamp can be obtained by combining the target three-dimensional optical flow information. For example, if t represents the initial timestamp and t3 represents the specified timestamp, the difference timestamp dt can be represented as t3-t.

[0126] The position information of each pixel in the acquisition device coordinate system of a specified timestamp can be obtained by the following formula (4).

[0127]

[0128] Among them, P w,t+dt That is, the predicted position information of each pixel in the coordinate system of the acquisition device.

[0129] Furthermore, it is necessary to convert the position information of each pixel in the acquisition device coordinate system to the image coordinate system. Exemplarily, the conversion can be performed in the form of a projection function using the following formula (5).

[0130]

[0131] Among them, [u ′ ,v ′ ] is the predicted position information of each pixel in the image coordinate system at the specified timestamp, thus combining the [u ′ ,v ′ ] to get the target image. Among them, Project is the projection function.

[0132] From the foregoing introduction, it can be seen that the embodiment of the present application predicts the target image based on the target three-dimensional optical flow information of each pixel point in the real scene image. Compared with predicting the real scene image as a whole, the obtained target image has a higher accuracy.

[0133] Optionally, if the target image is predicted directly by the mathematical model introduced above, there may be problems such as noise and unevenness that are difficult to process. Therefore, in other embodiments, the target image can also be obtained by a deep learning method. Specifically, step S240 can also include step S241 and step S242.

[0134] Step S241: taking the difference between the designated timestamp and the initial timestamp as a differential timestamp.

[0135] Step S242: taking the target three-dimensional optical flow information and the difference timestamp as input quantities and inputting them into a pre-trained deep learning model, and obtaining the output quantity of the deep learning model as the target image.

[0136] The difference timestamp is the difference between the specified timestamp and the initial timestamp. Furthermore, the target 3D optical flow information and the difference timestamp can be used as inputs to a pre-trained deep learning model, thereby obtaining the output of the deep learning model as the target image.

[0137] For example, please refer to Figures 5 to 7 ,in, Figure 5 A schematic diagram of a real scene image provided by an embodiment of the present application is shown. Figure 6 A schematic diagram showing target three-dimensional optical flow information provided by an embodiment of the present application is shown; Figure 7 A schematic diagram of a target image provided by an embodiment of the present application is shown.

[0138] in, Figure 5 The real scene image 510 in the image is obtained by processing the above steps. Figure 6 The target three-dimensional optical flow information 610 shown is further predicted for the real scene image 510 to obtain Figure 7 The target image 710 with a specified timestamp in FIG.

[0139] Step S250: Displaying the target image through the augmented reality device.

[0140] Among them, step S250 has been introduced in detail in the above embodiment and will not be repeated here.

[0141] The blind spot display method provided in the embodiment of the present application first determines the visual field blind spot corresponding to the target person based on the augmented reality device worn by the target person in the target vehicle; then obtains the real scene image corresponding to the visual field blind spot; then obtains the target three-dimensional optical flow information of each pixel point in the real scene image based on the real scene image and the pre-acquired specified external parameters, wherein the specified external parameters are the external parameters corresponding to the acquisition device used to obtain the real scene image, and the target three-dimensional optical flow information includes the first position information and the first speed information of each pixel point in the world coordinate system; based on the target three-dimensional optical flow information, the initial timestamp and the specified timestamp, the predicted target image is generated; thereby the target image is displayed through the augmented reality device. In this embodiment, the position information and speed information can be determined for different areas respectively to improve the accuracy of the speed information of the acquired real scene image. In addition, the target three-dimensional optical flow information is obtained by continuous frame smoothing, which improves the accuracy of the acquired target three-dimensional optical flow information, thereby improving the effect of the target image obtained by subsequent prediction to a certain extent. In addition, the embodiment of the present application predicts the target image based on the target three-dimensional optical flow information of each pixel point in the real scene image. Compared with predicting the real scene image as a whole, the obtained target image has a higher accuracy.

[0142] See also Figure 8 , Figure 8 A method flow chart of a blind area display method provided by an embodiment of the present application is shown. The blind area display method can be applied to Figure 1 In the blind spot display scenario shown in , the method may specifically include steps S310 to S3100.

[0143] Step S310: Determine the eye posture information of the target person based on the augmented reality device.

[0144] Step S320: Calculate the area corresponding to the obstructing object in the target vehicle through a perspective projection algorithm based on the posture information as the blind spot.

[0145] It can be seen from the introduction of the aforementioned embodiments that the position information of the target person's glasses can be obtained through an augmented reality device. Among them, the augmented reality device can be configured with a camera, so that the augmented reality device can realize simultaneous positioning and mapping (SLAM) through the camera, and then determine the position of the augmented reality device in the vehicle coordinate system, and further obtain the position information of the eyes of the target person wearing the augmented reality device. It is understandable that the position information of the eyes of the determined target person is characterized in the vehicle coordinate system. Exemplarily, simultaneous positioning and mapping can be realized by the Structure from Motion (SFM) algorithm, and specifically, COLMAP (COLLISION-MAPPING) software can be used to realize simultaneous positioning and mapping.

[0146] It can be seen from the introduction of the above embodiments that there is an obstructing object in the target vehicle, and the target person's line of sight will form a blind spot after passing through the obstructing object. Exemplarily, based on the acquired eye posture information, the area corresponding to the obstructing object in the target vehicle can be calculated according to the perspective projection algorithm as the blind spot. In the embodiment provided in the present application, the obstructing object can be the A-pillar of the target vehicle.

[0147] Step S330: Acquire a real scene image corresponding to the blind spot of the visual field.

[0148] Among them, step S330 has been introduced in detail in the above embodiment and will not be repeated here.

[0149] Step S340: predicting the historical real scene images collected in advance, and displaying the predicted historical real scene images in the augmented reality device.

[0150] Step S350: Determine the historical timestamp of the historical real scene image collected.

[0151] Step S360: Determine an intermediate timestamp for displaying the predicted real scene image in the augmented reality device.

[0152] Step S370: taking the difference between the intermediate timestamp and the historical timestamp as the designated timestamp.

[0153] As can be seen from the above introduction, the designated timestamp is the timestamp after the initial timestamp. It is understandable that the specific value of the designated timestamp will affect the effect of the target image finally predicted and generated. For some embodiments, the designated timestamp can be determined by combining the historical timestamp of the historical real scene image collected and the intermediate timestamp of the predicted real scene image displayed in the augmented reality device.

[0154] Among them, the historical real-scene image can be the previous frame image of the real-scene image, or it can be a frame image corresponding to N frames before the real-scene image, where N is an integer greater than 1, and the embodiment of the present application does not make specific limitations.

[0155] It is understandable that the historical real scene image is collected before the real scene image is acquired, so that the historical real scene image can be directly predicted at this time, and the predicted historical real scene image can be displayed in the augmented reality device.

[0156] Furthermore, the historical timestamp of the acquisition of the historical real scene image and the intermediate timestamp of the display of the predicted real scene image in the augmented reality device can be determined respectively. It can be understood that the difference between the intermediate timestamp and the historical timestamp can be regarded as the delay time required for acquiring the image, predicting the image and then displaying it, so that the difference between the intermediate timestamp and the historical timestamp can be used as the specified timestamp.

[0157] For example, if the historical timestamp of the collected historical real scene image is T0; then the prediction starts at timestamp T1; the prediction ends at timestamp T2, and the predicted historical real scene image is obtained; further, the screen of the augmented reality device is waited to be refreshed, so that it is displayed at the intermediate timestamp T3. Finally, the historical real scene image collected at the historical timestamp T0 will be seen by the user at the intermediate timestamp T3 after processing. Thus, the specified timestamp can be obtained as T3-T0.

[0158] Optionally, since the refresh rate and refresh time of the screen are known, and the timestamp at which the image is expected to be displayed in the augmented reality device is the intermediate timestamp T3, and the time required to predict a frame of image is T2-T1, the time T3-(T2-T1) can be set to start predicting the captured real-scene image, so that the target image can be subsequently displayed in the augmented reality device at the intermediate timestamp T3.

[0159] Step S380: predicting a target image of a specified timestamp based on the real scene image, where the specified timestamp is a timestamp after an initial timestamp, and the initial timestamp is a timestamp when the real scene image is acquired.

[0160] Among them, step S380 has been introduced in detail in the above embodiment and will not be repeated here.

[0161] Step S390: performing image processing on the target image to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising.

[0162] Step S3100: Displaying the target image after image processing through the augmented reality device.

[0163] After acquiring the target image, the target image may be further processed to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising.

[0164] In some implementations, the acquired target image may be sent to an augmented reality device, and then the target image may be processed by an image processing module configured in the augmented reality device.

[0165] In other embodiments, after acquiring the target image, the target image may be directly processed by an image processing module in the vehicle-mounted system of the target vehicle, and then the processed target image may be sent to the augmented reality device, so that the augmented reality device can directly display the acquired processed target image.

[0166] The blind spot display method provided in the embodiment of the present application determines the posture information of the eyes of the target person based on the augmented reality device; calculates the area corresponding to the occluding object in the target vehicle as the blind spot of the field of view through a perspective projection algorithm based on the posture information; obtains the real-scene image corresponding to the blind spot of the field of view; predicts the historical real-scene image collected in advance, and displays the predicted historical real-scene image in the augmented reality device; determines the historical timestamp of collecting the historical real-scene image; determines the intermediate timestamp of displaying the predicted real-scene image in the augmented reality device; uses the difference between the intermediate timestamp and the historical timestamp as the designated timestamp; predicts the target image of the designated timestamp based on the real-scene image, the designated timestamp is the timestamp after the initial timestamp, and the initial timestamp is the timestamp when the real-scene image is obtained; performs image processing on the target image to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising; and displays the image-processed target image through the augmented reality device. The embodiment of the present application determines the designated timestamp through the historical real scene image, and then generates the target image based on the designated timestamp, which can improve the accuracy of the acquired target image and improve the effect of the subsequent display of the target image. In addition, instead of directly displaying the target image in the augmented reality device, the target image is also processed, and then the processed target image is displayed in the augmented reality device, thereby further improving the display effect.

[0167] See also Fig. 9 , Fig. 9A structural block diagram of a blind spot display device 900 provided in an embodiment of the present application is shown. The blind spot display device 900 includes: a determination unit 910, an acquisition unit 920, a prediction unit 930 and a display unit 940.

[0168] The determination unit 910 is used to determine the blind spot of the field of vision corresponding to the target person in the target vehicle based on the augmented reality device worn by the target person.

[0169] Optionally, the determination unit 910 can also be used to predict the historical real-scene image collected in advance, and display the predicted historical real-scene image in the augmented reality device; determine the historical timestamp of collecting the historical real-scene image; determine the intermediate timestamp of displaying the predicted real-scene image in the augmented reality device; and use the difference between the intermediate timestamp and the historical timestamp as the designated timestamp.

[0170] Optionally, the determination unit 910 can also be used to determine the posture information of the eyes of the target person based on the augmented reality device; and calculate the area corresponding to the occluded object in the target vehicle through a perspective projection algorithm based on the posture information as the blind spot of the field of vision.

[0171] The acquisition unit 920 is used to acquire the real scene image corresponding to the blind spot of the visual field.

[0172] The prediction unit 930 is configured to predict a target image at a specified timestamp based on the real scene image, where the specified timestamp is a timestamp after an initial timestamp, and the initial timestamp is a timestamp when the real scene image is acquired.

[0173] Optionally, the prediction unit 930 can also be used to obtain target three-dimensional optical flow information of each pixel point in the real scene image based on the real scene image and pre-acquired specified external parameters, wherein the specified external parameters are external parameters corresponding to the acquisition device used to obtain the real scene image, and the target three-dimensional optical flow information includes the first position information and first speed information of each pixel point in the world coordinate system; based on the target three-dimensional optical flow information, the initial timestamp and the specified timestamp, a predicted target image is generated.

[0174] Optionally, the prediction unit 930 can also be used to obtain the movement speed of the target vehicle; based on the real scene image and the specified external parameters, determine the two-dimensional optical flow information of each pixel point in the real scene image, wherein the two-dimensional optical flow information includes the second position information and second speed information of each pixel point in the image coordinate system, and the image coordinate system is a coordinate system established based on the real scene image; based on the specified external parameters, the movement speed and the two-dimensional optical flow information, determine the third position information and third speed information of each pixel point in the three-dimensional vector field in the world coordinate system, wherein the pixels included in the three-dimensional vector field correspond to the pixels included in the real scene image; based on the third position information and the third speed information, determine the target three-dimensional optical flow information.

[0175] Optionally, the prediction unit 930 can also be used to obtain fourth position information of the real scene corresponding to the real scene image and fourth speed information of the real scene relative to the target vehicle, wherein the fourth position information and the fourth speed information are both in the world coordinate system; based on the specified external parameter, the fourth speed information is converted into fifth speed information in the acquisition device coordinate system, and based on the specified external parameter, the fourth position information is converted into fifth position information in the acquisition device coordinate system; based on the projection function, the fifth speed information is converted into second speed information in the image coordinate system, and based on the projection function, the fifth position information is converted into second position information in the image coordinate system; the second speed information and the second position information are used as the two-dimensional optical flow information.

[0176] Optionally, the prediction unit 930 can also be used to determine the position information of the first real scene corresponding to the first image area and the position information of the second real scene corresponding to the second image area in the world coordinate system, respectively, wherein the first image area is the area including the target object in the real scene image, and the second image area is the area not including the target object in the real scene image; use the position information of the first real scene and the position information of the second real scene as the fourth position information; obtain the speed information of the first real scene corresponding to the first image area collected by the target vehicle; use the movement speed of the target vehicle as the speed information of the second real scene corresponding to the second image area, and use the speed information of the first real scene and the speed information of the second real scene as the fourth speed information.

[0177] Optionally, the prediction unit 930 may also be configured to use the third position information and the third speed information as initial three-dimensional optical flow information; and perform continuous frame smoothing processing on the initial three-dimensional optical flow information to obtain the target three-dimensional optical flow information.

[0178] Optionally, the prediction unit 930 can also be used to take the difference between the specified timestamp and the initial timestamp as a differential timestamp; take the target three-dimensional optical flow information and the differential timestamp as input, input them into a pre-trained deep learning model, and obtain the output of the deep learning model as the target image.

[0179] The display unit 940 is configured to display the target image via the augmented reality device.

[0180] Optionally, the display unit 940 can also be used to perform image processing on the target image to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising; and the target image after image processing is displayed through the augmented reality device.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0182] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] See also Fig.10 , Fig.10 The structural block diagram of an electronic device 1000 provided in an embodiment of the present application is shown. The electronic device 1000 may be a vehicle system, which may be arranged in a vehicle. The electronic device 1000 in the present application may include one or more of the following components: a processor 1011, a memory 1012, and one or more application programs, wherein the processor 1011 is electrically connected to the memory 1012, and the one or more program configurations are used to execute the method described in each embodiment of the blind spot display method as described above.

[0184] The processor 1011 may include one or more processing cores. The processor 1011 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1012, and calling data stored in the memory 1012. Optionally, the processor 1011 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 1011 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and computer programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 1011, but may be implemented by a communication chip alone. Specifically, the method described in the above-mentioned embodiment may be executed by one or more processors 1011.

[0185] For some embodiments, the memory 1012 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1012 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1012 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 1000 during use, etc.

[0186] See also Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1100 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0187] The computer readable storage medium 1100 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1100 has storage space for program codes 1110 that perform any of the method steps in the above method. These program codes may be read from or written to one or more computer program products. The program code 1110 may be compressed, for example, in an appropriate form.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A blind area display method, characterized in that: include: Determine a blind spot in the field of vision corresponding to a target person in the target vehicle based on an augmented reality device worn by the target person in the target vehicle; Acquire a real scene image corresponding to the blind area of ​​the visual field; Predicting a target image at a specified timestamp based on the real scene image, where the specified timestamp is a timestamp after an initial timestamp, and the initial timestamp is a timestamp when the real scene image is acquired; The target image is displayed through the augmented reality device.

2. The method according to claim 1, characterized in that The predicting a target image with a specified timestamp based on the real scene image comprises: Based on the real scene image and the pre-acquired specified external parameters, the target three-dimensional optical flow information of each pixel in the real scene image is obtained, wherein the specified external parameters are external parameters corresponding to the acquisition device used to obtain the real scene image, and the target three-dimensional optical flow information includes the first position information and the first speed information of each pixel in the world coordinate system; A predicted target image is generated based on the target three-dimensional optical flow information, the initial timestamp, and the designated timestamp.

3. The method according to claim 2, characterized in that The step of acquiring target three-dimensional optical flow information of each pixel in the real scene image based on the real scene image and the pre-acquired specified external parameters includes: Obtaining the moving speed of the target vehicle; Based on the real scene image and the specified external parameters, determine the two-dimensional optical flow information of each pixel in the real scene image, wherein the two-dimensional optical flow information includes second position information and second speed information of each pixel in an image coordinate system, and the image coordinate system is a coordinate system established based on the real scene image; Based on the specified extrinsic parameter, the motion speed and the two-dimensional optical flow information, determine third position information and third speed information of each pixel point in the three-dimensional vector field in the world coordinate system, wherein the pixel points included in the three-dimensional vector field correspond to the pixel points included in the real scene image; The target three-dimensional optical flow information is determined based on the third position information and the third speed information.

4. The method according to claim 3, characterized in that The determining, based on the real scene image and the specified external parameter, the two-dimensional optical flow information of each pixel in the real scene image includes: Acquire fourth position information of a real scene corresponding to the real scene image and fourth speed information of the real scene relative to the target vehicle, wherein the fourth position information and the fourth speed information are both in a world coordinate system; Converting the fourth speed information into fifth speed information in the acquisition device coordinate system based on the specified external parameter, and converting the fourth position information into fifth position information in the acquisition device coordinate system based on the specified external parameter; converting the fifth speed information into second speed information in the image coordinate system based on a projection function, and converting the fifth position information into second position information in the image coordinate system based on a projection function; The second speed information and the second position information are used as the two-dimensional optical flow information.

5. The method according to claim 4, characterized in that The real scene includes a first real scene and a second real scene, and the acquiring fourth position information of the real scene corresponding to the real scene image and fourth speed information of the real scene relative to the target vehicle includes: Determine respectively in a world coordinate system the position information of a first real scene corresponding to a first image region and the position information of a second real scene corresponding to a second image region, wherein the first image region is a region of the real scene image that includes the target object, and the second image region is a region of the real scene image that does not include the target object; Using the position information of the first real scene and the position information of the second real scene as the fourth position information; Acquire speed information of a first real scene corresponding to a first image area captured by the target vehicle; The moving speed of the target vehicle is used as the speed information of the second real scene corresponding to the second image area, and the speed information of the first real scene and the speed information of the second real scene are used as the fourth speed information.

6. The method according to claim 3, characterized in that: The determining the target three-dimensional optical flow information based on the third position information and the third speed information includes: Using the third position information and the third speed information as initial three-dimensional optical flow information; The initial three-dimensional optical flow information is subjected to continuous frame smoothing processing to obtain the target three-dimensional optical flow information.

7. The method according to claim 2, characterized in that The generating a predicted target image based on the target three-dimensional optical flow information, the initial timestamp and the specified timestamp includes: Taking the difference between the specified timestamp and the initial timestamp as a differential timestamp; The target three-dimensional optical flow information and the difference timestamp are used as input quantities and input into a pre-trained deep learning model to obtain an output quantity of the deep learning model as the target image.

8. The method according to claim 1, characterized in that Before predicting the target image with a specified timestamp based on the real scene image, the method further includes: Predicting the historical real scene images collected in advance, and displaying the predicted historical real scene images in the augmented reality device; Determine the historical timestamp of acquiring the historical real scene image; Determining an intermediate timestamp for displaying the predicted real scene image in the augmented reality device; The difference between the intermediate timestamp and the historical timestamp is used as the designated timestamp.

9. The method according to claim 1, characterized in that: The step of determining a blind spot corresponding to a target person based on an augmented reality device worn by the target person in the target vehicle includes: Determining the position information of the eyes of the target person based on the augmented reality device; Based on the posture information, an area corresponding to the obstructing object in the target vehicle is calculated by a perspective projection algorithm as the blind spot.

10. The method according to claim 1, characterized in that The displaying of the target image by the augmented reality device comprises: Performing image processing on the target image to obtain a processed target image, wherein the image processing includes at least one of image enhancement and image denoising; The target image after image processing is displayed through the augmented reality device.

11. A blind spot display device, characterized in that: include: A determination unit, configured to determine a blind spot of vision corresponding to a target person in a target vehicle based on an augmented reality device worn by the target person; An acquisition unit, used for acquiring a real scene image corresponding to the blind area of ​​the visual field; A prediction unit, configured to predict a target image of a specified timestamp based on the real scene image, wherein the specified timestamp is a timestamp after an initial timestamp, and the initial timestamp is a timestamp when the real scene image is acquired; A display unit is used to display the target image through the augmented reality device.

12. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.

Images