Blind spot image display method and apparatus, device, and storage medium

Abstract

Disclosed in embodiments of the present application are a blind spot image display method and apparatus, a device, and a storage medium, for use in adaptively determining the size of a display window used for displaying a blind spot image, so as to better assist a driver in identifying a driving risk in time. The method in the embodiments of the present application comprises: acquiring a blind spot image collected by a camera mounted on a vehicle; determining a window size on the basis of information about an obstacle around the vehicle, wherein a traffic risk level corresponding to the obstacle is positively correlated with the window size; and displaying the blind spot image on a screen by means of a display window corresponding to the window size. In this way, when a traffic risk level is high, a blind spot image can be displayed by means of a large display window, and when the traffic risk level is low, a blind spot image can be displayed by means of a small display window, so that the size of a display window used for displaying a blind spot image can be adaptively determined, and thus a driver can better view the blind spot image by means of a display window of a suitable size.

Description

A method, apparatus, device and storage medium for displaying images in blind spots

[0001] This application claims priority to Chinese Patent Application No. 202410922933.X, filed on July 10, 2024, entitled "A Method, Apparatus, Device and Storage Medium for Displaying Blind Spot Images", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent driving, specifically to a blind spot image display method, device, equipment, and storage medium. Background Technology

[0003] In current vehicles with intelligent driving capabilities, cameras may be installed at designated locations to capture blind spot images that the driver cannot see. When a trigger condition is met, the blind spot images captured by the camera will be displayed on the vehicle's screen through a display window to help the driver judge the traffic conditions to the rear and side and avoid collisions when turning or changing lanes.

[0004] However, when obstacles are close to the vehicle, the blind spot image displayed in the display window cannot help the driver to identify driving risks in a timely manner. Summary of the Invention

[0005] This application provides a blind spot image display method, apparatus, device, and storage medium. Based on information about obstacles around the vehicle, the size of the display window for displaying the blind spot image is adaptively determined, so that the driver can better view the blind spot image through a display window of a reasonable size, thereby better assisting the driver in timely identification of driving risks.

[0006] In view of this, in a first aspect, embodiments of this application provide a blind spot image display method, the method comprising: acquiring a blind spot image captured by a camera; the camera being mounted on a vehicle; determining a window size based on obstacle information around the vehicle; the passage hazard level corresponding to the obstacle being positively correlated with the window size; and displaying the blind spot image on a screen through a display window corresponding to the window size.

[0007] In the method provided by this application, the traffic hazard level corresponding to the obstacle is positively correlated with the window size. Thus, based on the obstacle information around the vehicle, when the traffic hazard level is high, such as when the obstacle is close to the vehicle, a larger display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too small to see the obstacle clearly. When the traffic hazard level is low, such as when the obstacle is far from the vehicle, a smaller display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too large and affects the driver's use or viewing of navigation information. This achieves adaptive determination of the display window size for displaying the blind spot image, so that the driver can better view the blind spot image through a display window of a reasonable size, thereby better assisting the driver in timely identification of driving risks.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the window size includes: determining the distribution location information of obstacles in the blind spot image display area based on obstacle information around the vehicle; and determining the window size based on the distribution location information. In this application's implementation, by defining the blind spot image display area and determining the distribution location information of obstacles in the blind spot image display area, the size of the display window used to display the blind spot image is adaptively determined, thereby better assisting the driver in timely identification of driving risks.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the window size based on the distribution location information includes: if the obstacle is located in a first area of ​​the blind spot image display area, determining the window size to a first preset value; if the obstacle is located in a second area of ​​the blind spot image display area, determining the window size to a second preset value; wherein, the traffic hazard level corresponding to the obstacle being located in the first area is greater than the traffic hazard level corresponding to the obstacle being located in the second area, and the first preset value is greater than the second preset value. In this application embodiment, by dividing the blind spot image display area, different window sizes are achieved for obstacles located in different areas, thereby adaptively determining the size of the display window used to display the blind spot image, thus better assisting the driver in timely identification of driving risks.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the distribution location information includes: determining the relative relationship between the vehicle and the obstacle based on blind spot images, vehicle information, and obstacle information around the vehicle; and determining the distribution location information based on the relative relationship. In this application's implementation, combining multiple pieces of information allows for a more accurate determination of the obstacle's distribution location information, thus facilitating a more accurate subsequent determination of the window size.

[0011] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: determining a first region and a second region based on the vehicle's performance; the traffic hazard level corresponding to an obstacle located in the first region is greater than the traffic hazard level corresponding to an obstacle located in the second region, and the distance between the first region and the vehicle is less than the distance between the second region and the vehicle; and constructing a blind spot image display area based on the first region and the second region. In this embodiment of the application, corresponding blind spot image display areas can be constructed based on the performance of different vehicles, thus adapting the blind spot image display area to the vehicle and facilitating more accurate determination of the window size subsequently.

[0012] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the window size includes: determining the relative relationship between the vehicle and the obstacles based on obstacle information around the vehicle; determining the window size based on the relative relationship and a mapping relationship; the mapping relationship is used to characterize the relationship between the relative relationship and the window size. In this application embodiment, by defining a mapping relationship, after determining the relative relationship between the vehicle and the obstacles, the size of the display window used to display blind spot images can be adaptively determined in conjunction with the mapping relationship, thereby better assisting the driver in timely identification of driving risks.

[0013] In conjunction with the first aspect, in one possible implementation of the first aspect, the relative relationship includes the relative distance between the vehicle and the obstacle; the mapping relationship includes: if the relative distance is greater than or equal to a first distance threshold, the window size is a third preset value; if the relative distance is greater than a second distance threshold and less than the first distance threshold, the window size is a fourth preset value; the fourth preset value is greater than the third preset value; if the relative distance is less than or equal to the second distance threshold, the window size is a fifth preset value; the fifth preset value is greater than the fourth preset value. In this application's implementation, by using different magnitude relationships between the relative distance between the vehicle and the obstacle and the distance threshold, different magnitude relationships correspond to different window sizes. This facilitates the subsequent adaptive determination of the display window size for displaying blind spot images based on the mapping relationship, thereby better assisting the driver in timely identification of driving risks.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the relative relationship includes: the relative distance between the vehicle and the obstacle, and the relative speed between the vehicle and the obstacle; determining the window size based on the relative relationship and the mapping relationship includes: determining the weight corresponding to the relative speed; updating the relative distance based on the weight to obtain the updated relative distance; and determining the window size based on the updated relative distance and the mapping relationship. In this embodiment of the application, the window size is determined by combining the relative speed between the vehicle and the obstacle and the weight corresponding to the relative speed between the vehicle and the obstacle. This takes into account the influence of relative speed and relative distance on the degree of traffic hazard, and can more accurately determine the corresponding window size.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the window size includes: inputting the blind spot image into a window recognition model to identify obstacle information around the vehicle and determining the window size; the window recognition model is trained using blind spot training data and window size annotation information corresponding to the blind spot training data. In this embodiment of the application, by pre-training the window recognition model, the model can learn the relationship between the blind spot image and the window size. Thus, after inputting the blind spot image into the window recognition model, it can adaptively determine the size of the display window used to display the blind spot image, thereby better assisting the driver in timely identification of driving risks.

[0016] In conjunction with the first aspect, in one possible implementation of the first aspect, the blind spot image is input into the window recognition model to identify obstacle information around the vehicle and determine the window size. This includes: inputting the blind spot image and vehicle information into the window recognition model to identify obstacle information around the vehicle and determine the window size; the vehicle information includes the vehicle's speed and direction of travel. In this embodiment of the application, the window size is determined by combining the blind spot image, the vehicle's speed, and the direction of travel. This takes into account the influence of the vehicle's own motion state on the window size, and can more accurately and reasonably determine the corresponding window size.

[0017] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: training an initial model based on blind spot training data, window size annotation information, vehicle speed training data corresponding to the blind spot training data, and driving direction training data to obtain a loss value; and iteratively training the initial model based on the loss value to obtain a window recognition model. In this embodiment, by training the window recognition model using blind spot training data, window size annotation information, vehicle speed training data, and driving direction training data, the window recognition model can learn the influence of information such as the correlation and continuity of obstacles between frames in the blind spot training data on the window size, as well as the influence of different vehicle speeds and driving directions on the window size. This allows the model to learn the relationship between different blind spot images, vehicle speeds, driving directions, and window sizes, facilitating the subsequent adaptive determination of the display window size for displaying blind spot images based on the window recognition model, thereby better assisting the driver in timely identification of driving risks.

[0018] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: determining the relative relationship between the vehicle and the obstacle based on the blind spot image, vehicle information, and obstacle information around the vehicle; and adjusting the viewing angle of the blind spot image based on the relative relationship. In this application's implementation, by using multiple pieces of information, the relative relationship between the vehicle and the obstacle can be determined more accurately, enabling adaptive adjustment of the viewing angle when the blind spot image is displayed, so that the displayed blind spot image can better assist the driver in timely identifying driving risks.

[0019] In conjunction with the first aspect, in one possible implementation of the first aspect, the observation perspective includes a field of view and an observation focus. The method further includes: cropping the blind spot image according to the field of view and the observation focus to obtain a cropped blind spot image; displaying the blind spot image includes: displaying the cropped blind spot image on the screen through a display window of a window size corresponding to the screen size. In this embodiment of the application, the blind spot image is cropped accordingly by the field of view and the observation focus, reducing the content in the original blind spot image, so that the cropped blind spot image can be more focused on the obstacle, and the displayed cropped blind spot image can better assist the driver in timely identification of driving risks.

[0020] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the window size includes: determining the window size based on obstacle information around the vehicle when a trigger condition is met; the trigger condition is at least one of the following: the vehicle's turn signal is activated, the vehicle is turning, or the vehicle is changing lanes. In this application's implementation, determining the window size based on obstacle information around the vehicle avoids immediately displaying blind spot images upon meeting the trigger condition, thus preventing the displayed blind spot images from containing obstacles, thereby better assisting the driver.

[0021] Secondly, this application provides a blind spot image display device, the device comprising:

[0022] The acquisition module acquires blind spot images captured by the camera; the camera is mounted on the vehicle.

[0023] The determination module is used to determine the window size based on information about obstacles around the vehicle; the passage hazard level corresponding to the obstacle is positively correlated with the window size.

[0024] The display module is used to display blind spot images on the screen through a display window of a corresponding window size.

[0025] The blind spot image display device has the function of implementing the blind spot image display method in the first aspect or any possible embodiment of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the above function.

[0026] The beneficial effects shown in this aspect are similar to those in the first aspect, as detailed in the first aspect, and will not be repeated here.

[0027] Thirdly, this application provides a blind spot image display device, which may include: a memory for storing instructions; and a processor for executing the instructions in the memory to perform the blind spot image display method in the first aspect or any optional embodiment of the first aspect.

[0028] Fourthly, this application provides a computer-readable storage medium that may include instructions that, when executed on a computer, cause the computer to perform the blind spot image display method in the first aspect or any optional embodiment of the first aspect.

[0029] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the blind spot image display method in the first aspect or any optional embodiment of the first aspect.

[0030] Sixthly, this application provides a chip system including a processor for supporting a device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the device. This chip system may be composed of chips or may include chips and other discrete devices.

[0031] In a seventh aspect, embodiments of this application provide a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the blind spot image display method in the first aspect or any optional embodiment of the first aspect.

[0032] Eighthly, this application provides a vehicle that may include: a memory for storing instructions; and a processor for executing the instructions in the memory to perform the blind spot image display method in the first aspect or any optional embodiment of the first aspect. Attached Figure Description

[0033] Figure 1 is a schematic diagram of the architecture of a blind spot image display system provided in an embodiment of this application;

[0034] Figure 2 is a flowchart illustrating a blind spot image display method provided in an embodiment of this application;

[0035] Figure 3 is a schematic diagram of the screen and display window provided in an embodiment of this application;

[0036] Figure 4 is a flowchart illustrating another blind spot image display method provided in an embodiment of this application;

[0037] Figure 5 is a schematic diagram of a blind spot image display area provided in an embodiment of this application;

[0038] Figure 6 is a schematic diagram showing the relationship between an observation angle and the distribution of obstacles according to an embodiment of this application;

[0039] Figure 7 is a flowchart illustrating another blind spot image display method provided in an embodiment of this application;

[0040] Figure 8 is a schematic diagram of a mapping function provided in an embodiment of this application;

[0041] Figure 9 is a flowchart illustrating another blind spot image display method provided in an embodiment of this application;

[0042] Figure 10 is a schematic diagram of the composition structure of a blind spot image display device provided in an embodiment of this application;

[0043] Figure 11 is a structural schematic diagram of a blind spot image display device provided in an embodiment of this application;

[0044] Figure 12 is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0047] The method provided in this application embodiment can be applied to scenarios where a camera assists users in viewing blind spot risks. When the level of traffic danger is high, a larger display window is used to display the blind spot image, and when the level of traffic danger is low, a smaller display window is used to display the blind spot image.

[0048] Before describing in detail the blind spot image display method provided in the embodiments of this application, the architecture of a blind spot image display system provided in the embodiments of this application will be described first. Please refer to Figure 1, which is a schematic diagram of the architecture of a blind spot image display system provided in the embodiments of this application. The blind spot image display system may include a camera, a screen, and a processor installed on the vehicle. The camera collects blind spot images, the processor acquires the blind spot images, and determines the window size based on obstacle information around the vehicle. When the traffic hazard level corresponding to the obstacle is high, the blind spot image is displayed on the screen through a larger display window; when the traffic hazard level corresponding to the obstacle is low, the blind spot image is displayed on the screen through a smaller display window. This adaptive determination of the corresponding display window size for displaying blind spot images allows the driver to better view the blind spot images through a display window of a reasonable size, thereby assisting the driver in timely identification of driving risks.

[0049] Specifically, please refer to Figure 2, which is a flowchart illustrating a blind spot image display method provided in this application. This blind spot image display method can be executed by a blind spot image display device. The blind spot image display method provided in this application embodiment may include:

[0050] 201. Obtain blind spot images captured by the camera.

[0051] The camera is installed on the vehicle. The camera can be a bullet camera, a square camera, a spherical camera, etc. The camera can be installed on the side of the vehicle, such as near the rearview mirror, near the front wheel, etc.

[0052] In this application embodiment, blind spot images refer to video stream data corresponding to the vehicle's surrounding environment that the driver cannot see.

[0053] 202. Determine the window size based on information about obstacles around the vehicle.

[0054] Among them, the passage hazard level corresponding to the obstacle is positively correlated with the window size, that is, the higher the passage hazard level corresponding to the obstacle, the larger the window size, and the lower the passage hazard level corresponding to the obstacle, the smaller the window size.

[0055] Obstacles around a vehicle can affect its passage. The traffic hazard level corresponding to an obstacle can be categorized as follows: the closer the obstacle is to the vehicle, the higher the traffic hazard level; the farther the obstacle is from the vehicle, the lower the traffic hazard level; if the obstacle is moving, the traffic hazard level is higher; if the obstacle is stationary, the traffic hazard level is lower; the larger the obstacle, the higher the traffic hazard level; the smaller the obstacle, the lower the traffic hazard level. This application does not limit the implementation method of the traffic hazard level corresponding to the obstacle in its embodiments.

[0056] Information on obstacles around a vehicle can include whether there are obstacles around the vehicle, the location of the obstacles, the type of obstacles, the size of the obstacles, the speed of the obstacles, the direction of the obstacles, and so on.

[0057] Blind spot images include information about the vehicle's surrounding environment, such as whether there are obstacles around the vehicle, the type and size of the obstacles, etc. In this embodiment, information about obstacles around the vehicle can be obtained by identifying blind spot images, or by processing data collected by vehicle sensors such as LiDAR.

[0058] The window size can be set to large, medium, small, or hidden, etc. It can also be set to specific numerical values ​​such as 1, 0.8, 0.6, 0, etc., or specific dimensions such as a1×b1, a2×b2, a3×b3, 0×0, etc. Specifically, if the window size is set to large, medium, small, or hidden, large, medium, and small will each correspond to a specific dimension; for example, large corresponds to a1×b1, medium to a2×b2, and small to a3×b3. If the window size is set to specific numerical values, 1, 0.8, 0.6, etc., will each correspond to a specific dimension; for example, 1 corresponds to a1×b1, 0.8 to a2×b2, 0.6 to a3×b3, etc.

[0059] 203. Display the blind spot image on the screen using a display window of the corresponding window size.

[0060] The screen in this embodiment can be an in-vehicle screen, such as a central control screen, or a display area corresponding to a Head-Up Display (HUD), such as the windshield or a certain distance in front of the driver. The HUD utilizes optical reflection to project driving assistance information, navigation information, blind spot images, etc., onto the windshield or a certain distance in front of the driver. This allows the driver to view relevant information without frequently looking down while driving, thus improving safety.

[0061] It should be noted that if the obstacle information around the vehicle indicates that there are no obstacles around the vehicle, or the location information of the obstacle differs significantly from the location information of the vehicle, it means that there are no obstacles to the side of the vehicle or the obstacle is far away from the vehicle, that is, there is no danger of passing. In this case, the window size will be set to not display, 0, etc., so there will be no display window on the screen and the blind spot image will not be displayed.

[0062] Please refer to Figure 3, which is a schematic diagram of the screen and display window provided in an embodiment of this application. The upper sub-figure of Figure 3 illustrates the screen and display window when the window size is large; the middle sub-figure of Figure 3 illustrates the screen and display window when the window size is small; the lower sub-figure of Figure 3 illustrates the screen when the window size is not displayed, in which case there is no display window.

[0063] The width and height of the display window corresponding to the window size can be determined based on information such as the screen of the specific product, the image resolution of the camera, and the image aspect ratio.

[0064] As can be seen, in the method provided by this application, the traffic hazard level corresponding to the obstacle is positively correlated with the window size. Thus, based on the obstacle information around the vehicle, when the traffic hazard level is high, such as when the obstacle is close to the vehicle, a larger display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too small to see the obstacle clearly. When the traffic hazard level is low, such as when the obstacle is far from the vehicle, a smaller display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too large and affects the driver's use or viewing of navigation information. This achieves adaptive determination of the size of the display window used to display the blind spot image, so that the driver can better view the blind spot image through a display window of a reasonable size, thereby better assisting the driver in timely identification of driving risks.

[0065] Please refer to Figure 4, which is a flowchart illustrating another blind spot image display method provided in this application embodiment. The blind spot image display method provided in this application embodiment may include:

[0066] 401. Based on the vehicle's performance, determine the first region and the second region.

[0067] Among them, the traffic hazard level corresponding to the obstacle being located in the first area is greater than the traffic hazard level corresponding to the obstacle being located in the second area, and the distance between the first area and the vehicle is less than the distance between the second area and the vehicle.

[0068] Vehicle performance can include: vehicle power mode, driving mode, and braking sensitivity, etc.

[0069] In this embodiment, the first region and the second region can also be determined jointly based on the camera's calibration parameters and the vehicle's performance. The camera's calibration parameters can include intrinsic and extrinsic parameters. Intrinsic parameters can include focal length, pixel size, etc., while extrinsic parameters can include the camera's mounting position, orientation, and rotation angle, etc.

[0070] 402. Based on the first and second regions, construct the blind spot image display area.

[0071] It is understood that, in the embodiments of this application, a third region and a fourth region may also be determined according to the actual situation, and then a blind spot image display area may be constructed based on the first region, the second region, the third region and the fourth region.

[0072] Please refer to Figure 5, which is a schematic diagram of a blind spot image display area provided in an embodiment of this application. The first area is a turning or lane-changing hazard area, and the traffic hazard level corresponding to an obstacle in the first area is high. The first area can be defined as a rectangular area of ​​3 meters * 5.5 meters, with a width of 3 meters extending laterally outward from each side of the vehicle and a length of 5.5 meters extending rearward from a line parallel to the driver's face and the front edge of the vehicle. The second area is a turning or lane-changing warning area, and the traffic hazard level corresponding to an obstacle in the second area is medium. The second area can be defined as a rectangular area of ​​3 meters * 67 meters, extending 67 meters rearward from the first area longitudinally. The third area is a turning or lane-changing caution area, and the traffic hazard level corresponding to an obstacle in the third area is low. The third area can be defined as a rectangular area of ​​2.6 meters * 72.5 meters, extending laterally outward from both the first and second areas together. It is understood that the above is only an illustrative description of the blind spot image display area and should not be construed as a limitation on the embodiments of this application.

[0073] It should be noted that if the obstacle is located outside the blind spot image display area, the corresponding passage hazard level of the obstacle is none, and the blind spot image does not need to be displayed on the screen, that is, there will be no display window on the screen.

[0074] In this embodiment, a corresponding blind spot image display area can be constructed based on the performance of different vehicles, so that the blind spot image display area is adapted to the vehicle, which makes it easier to determine the window size more accurately in the future.

[0075] 403. Obtain blind spot images captured by the camera.

[0076] It is understood that 403 is the same as 201 in the above embodiments, so it will not be described again.

[0077] 404. Obtain information about obstacles around the vehicle.

[0078] In this embodiment, the information on obstacles around the vehicle can be obtained by processing the data collected by vehicle-mounted sensors such as LiDAR, or by identifying blind spot images, etc.

[0079] 405. Based on information about obstacles around the vehicle, determine the distribution location information of obstacles in the blind spot image display area.

[0080] It is understood that the parts of 405 that are the same as those of 202 in the above embodiment will not be described again here.

[0081] Optionally, in this embodiment, 405 may include: determining the relative relationship between the vehicle and the obstacles based on obstacle information around the vehicle, and then determining the distribution location information of the obstacles in the blind spot image display area based on the relative relationship. The relative relationship between the vehicle and the obstacles may include: the relative distance between the vehicle and the obstacles, the relative orientation between the vehicle and the obstacles, the relative speed between the vehicle and the obstacles, etc. The relative orientation may include the specific angular orientation of the obstacle, such as whether it is in front of or behind the vehicle.

[0082] Optionally, embodiment 405 of this application may include: determining the relative relationship between the vehicle and obstacles based on obstacle information around the vehicle, combined with vehicle information and / or blind spot images; and then determining the distribution location information based on the relative relationship. This combination of multiple pieces of information allows for accurate determination of the obstacle distribution location information, facilitating a more accurate determination of the window size subsequently. Vehicle information may include vehicle position information, vehicle speed, vehicle direction of travel, etc. Vehicle information and obstacle information around the vehicle can be obtained by collecting relevant data through onboard sensors and using a neural network model on the vehicle to infer the collected data.

[0083] Optionally, in this embodiment, 405 may include: determining the relative relationship between the vehicle and the obstacles based on the obstacle information around the vehicle, combined with the camera calibration parameters, vehicle information and obstacle information around the vehicle, and then determining the distribution location information based on the relative relationship.

[0084] 406. Determine the window size based on the distribution location information.

[0085] In this embodiment, 406 may include: if the obstacle is located in a first area of ​​the blind spot image display area, determining the window size to a first preset value; if the obstacle is located in a second area of ​​the blind spot image display area, determining the window size to a second preset value. The traffic hazard level corresponding to the obstacle being located in the first area is greater than the traffic hazard level corresponding to the obstacle being located in the second area, and the first preset value is greater than the second preset value. By dividing the blind spot image display area in this way, different window sizes are corresponding to obstacles located in different areas, thereby adaptively determining the size of the display window used to display the blind spot image, which can better assist the driver in timely identification of driving risks.

[0086] Optionally, in this embodiment, the collision risk level between the obstacle and the vehicle can be determined based on the relative distance between the vehicle and the obstacle, the relative orientation between the vehicle and the obstacle, the vehicle speed, the vehicle's driving direction, the obstacle's speed, and the obstacle's direction of movement. Then, the window size is determined based on the distribution location information and the collision risk level. For example, if the obstacle is located at the edge of the first area and the obstacle's direction of movement is opposite to the vehicle's direction of movement, and there is no possibility of a future collision, the window size can be determined to be a sixth preset value. The sixth preset value can be less than the first preset value and greater than the second preset value, or the window size can be determined to be the second preset value. It should be understood that the above is only an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0087] In this embodiment, the window size can be determined based on obstacle information around the vehicle when a trigger condition is met. The trigger condition can be at least one of the following: turning on the vehicle's turn signal, the vehicle turning, or the vehicle changing lanes. Determining the window size by combining this with obstacle information around the vehicle avoids immediately displaying blind spot images upon meeting the trigger condition, which could result in the blind spot images not containing any obstacles, thus better assisting the driver.

[0088] Specifically, when the triggering conditions are met, the distribution location information of the obstacles in the blind spot image display area can be determined based on the obstacle information around the vehicle, and the window size can be determined based on the distribution location information.

[0089] 407. Adjust the viewing angle of blind spot images based on the relative relationship between the vehicle and the obstacle.

[0090] The relative relationship between the vehicle and the obstacle can include their relative orientation, while the viewing angle can include the field of view (FOV) and the focal point. This adaptive adjustment of the viewing angle during blind spot image display allows the displayed blind spot image to better assist the driver in timely identification of driving risks.

[0091] The field of view (FOV) should not exceed the camera's specifications. The focus of observation and the FOV size work together to ensure that obstacles are within the camera's field of view. Please refer to Figure 6, which is a schematic diagram illustrating the relationship between the viewing angle and obstacle distribution according to an embodiment of this application. The dashed line in the middle can be considered the focus of observation, and the angle formed by the dashed lines on the left and right sides can be considered the FOV size. When obstacles are located at the rear, the FOV can be 80 degrees, slightly towards the rear; when obstacles are located both at the front and rear, the FOV can be 100 degrees, a wide-angle view; when obstacles are located very close at the rear (e.g., <20cm), the FOV should include the edge of the vehicle body.

[0092] 408. Cropping the blind zone image according to the field of view and the observation focus yields the cropped blind zone image.

[0093] In this embodiment, the blind spot image is cropped by adjusting the field of view (FOV) and the focal point, reducing the content of the original blind spot image. This allows the cropped blind spot image to focus more on obstacles, thus better assisting the driver in timely identification of driving risks. For example, a frame of the blind spot image, such as a 1920x1080 resolution image, can be cropped and displayed. The center point of the cropped portion becomes the focal point. The FOV determines the size of the cropped image; for instance, cropping to 960x540 resolution reduces the FOV to half its original size. It is understood that the above is merely illustrative and should not be construed as limiting the embodiments of this application.

[0094] 409. Display the cropped blind spot image on the screen using a display window of the corresponding window size.

[0095] It is understood that 409 is similar to 203 in the above embodiment, except that 203 displays the blind spot image, while 409 displays the cropped blind spot image. Therefore, the same parts will not be described again here.

[0096] As can be seen, in the method provided by the embodiments of this application, by defining the blind spot image display area and determining the distribution location information of obstacles in the blind spot image display area, the size of the display window used to display the blind spot image is adaptively determined, thereby better assisting the driver to identify driving risks in a timely manner.

[0097] Please refer to Figure 7, which is a flowchart illustrating another blind spot image display method provided in this application embodiment. The blind spot image display method provided in this application embodiment may include:

[0098] 701. Obtain blind spot images captured by the camera.

[0099] It is understood that 701 is the same as 403 and 201 in the above embodiments, so it will not be described again.

[0100] 702. Obtain information about obstacles around the vehicle.

[0101] It is understood that 702 is the same as 404 in the above embodiments, so it will not be described again.

[0102] 703. Determine the relative relationship between the vehicle and the obstacles based on information about obstacles around the vehicle.

[0103] The relative relationship between the vehicle and the obstacle can include: the relative distance between the vehicle and the obstacle, the relative orientation between the vehicle and the obstacle, the relative speed between the vehicle and the obstacle, etc. The relative orientation can include the specific angular position of the obstacle, such as whether it is in front of or behind the vehicle.

[0104] It is understood that the method for determining the relative relationship between a vehicle and an obstacle in Embodiment 703 of this application is the same as the method for determining the relative relationship between a vehicle and an obstacle in Embodiment 405 above, and therefore will not be described again.

[0105] 704. Determine the window size based on relative and mapping relationships.

[0106] It is understood that the parts of 704 that are the same as those of 202 in the above embodiment will not be described again here.

[0107] Among them, the mapping relationship is used to represent the relationship between the relative relationship and the window size. The mapping relationship can be a mapping function, a mapping table, etc.

[0108] In this embodiment, the mapping relationship can specifically be used to characterize the relative distance between a vehicle and an obstacle. The mapping relationship can include: if the relative distance is greater than or equal to a first distance threshold, the window size is a third preset value; if the relative distance is greater than a second distance threshold but less than the first distance threshold, the window size is a fourth preset value, which is greater than the third preset value; if the relative distance is less than or equal to the second distance threshold, the window size is a fifth preset value, which is greater than the fourth preset value. By using different magnitude relationships between the relative distance between the vehicle and the obstacle and the distance threshold, different magnitude relationships correspond to different window sizes. This facilitates the subsequent adaptive determination of the display window size for displaying blind spot images based on the mapping relationship, better assisting the driver in timely identification of driving risks.

[0109] If there are multiple obstacles behind the vehicle, multiple relative distances will be determined. You can select the minimum value among these relative distances to determine the window size. In other words, you can select the relative distance corresponding to the obstacle closest to the vehicle to determine the window size.

[0110] Please refer to Figure 8, which is a schematic diagram of a mapping function provided in an embodiment of this application. In Figure 8, the minimum obstacle distance is the minimum value among multiple relative distances. Max in Figure 8 represents the first distance threshold, and Min represents the second distance threshold. The third setting value is 0, meaning that when the minimum value among multiple relative distances is greater than or equal to Max, it can be considered that there is no passage risk, and the blind spot image is not displayed; 0.5 < the fourth setting value < 1, meaning that the minimum value among multiple relative distances is less than Max and greater than Min, and the window size changes linearly from 0.5 to 1; the fifth setting value is 1, meaning that when the minimum value among multiple relative distances is less than or equal to Min, it can be considered that the passage risk is high, and the blind spot image can be displayed in the largest display window. It is understood that Figure 8 is an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0111] Optionally, in this embodiment, 704 may include: determining the weights corresponding to the relative speeds between the vehicle and the obstacle; updating the relative distance between the vehicle and the obstacle based on the weights to obtain the updated relative distance; and determining the window size based on the updated relative distance and the mapping relationship. By combining the relative speeds between the vehicle and the obstacle with the weights corresponding to those relative speeds, the window size is determined, taking into account the influence of relative speed and relative distance on the degree of traffic hazard, thus enabling a more accurate determination of the corresponding window size.

[0112] A higher relative speed allows for a smaller corresponding weight, and the relative distance can be obtained by multiplying the relative speed by the weight; conversely, a higher relative speed allows for a larger relative weight, and the relative distance can be obtained by dividing the relative speed by the weight. The weight can be a positive number greater than 0 and less than or equal to 1. This approach takes into account that higher relative speeds correspond to higher levels of road hazard. A smaller relative distance at higher relative speeds allows for a larger window size to be determined, thus better assisting drivers in identifying driving risks.

[0113] Optionally, in this embodiment, 704 may include: determining the weight corresponding to the size of the obstacle; updating the relative distance between the vehicle and the obstacle based on the weight to obtain the updated relative distance; and determining the window size based on the updated relative distance and the mapping relationship. By combining the relative speed between the vehicle and the obstacle, and the weight corresponding to the relative speed, to jointly determine the window size, the influence of relative speed and relative distance on the degree of traffic hazard is considered, enabling a more accurate determination of the corresponding window size.

[0114] The larger the obstacle, the smaller its corresponding weight can be set, and the relative distance can be obtained by multiplying the relative speed by the weight; conversely, the larger the obstacle, the larger its relative weight can be set, and the relative distance can be obtained by dividing the relative speed by the weight. The weight can be a positive number greater than 0 and less than or equal to 1. This approach takes into account that larger obstacles indicate a higher level of danger, and by setting a smaller relative distance for larger obstacles, a larger window size can be determined to better assist the driver in identifying driving risks.

[0115] This application embodiment can determine the window size based on relative and mapping relationships when triggering conditions are met. The triggering conditions can be at least one of the following: turning on the vehicle's turn signal, the vehicle turning, the vehicle changing lanes, etc.

[0116] 705. Adjust the viewing angle of blind spot images based on the relative relationship between the vehicle and the obstacle.

[0117] It is understood that 705 is the same as 407 in the above embodiments, so it will not be described again.

[0118] 706. Cropping the blind zone image according to the field of view and the observation focus yields the cropped blind zone image.

[0119] It is understood that 706 is the same as 408 in the above embodiments, so it will not be described again.

[0120] 707. Display the cropped blind spot image on the screen using a display window of the corresponding window size.

[0121] It is understood that 707 is the same as 409 in the above embodiment, so it will not be described again.

[0122] As can be seen, in the method provided by the embodiments of this application, by defining a mapping relationship, after determining the relative relationship between the vehicle and the obstacle, the size of the display window used to display the blind spot image can be adaptively determined in combination with the mapping relationship, so as to better assist the driver in timely identification of driving risks.

[0123] Please refer to Figure 9, which is a flowchart illustrating another blind spot image display method provided in this application embodiment. The blind spot image display method provided in this application embodiment may include:

[0124] 901. Train the initial model based on the blind zone training data and window size annotation information to obtain the loss value.

[0125] The blind spot training data can include historical blind spot images captured by cameras. These historical blind spot images refer to video stream data or images of the vehicle's surrounding environment that the driver could not see during historical driving. Window size annotation information can be obtained by manually annotating the blind spot training data or using annotation tools.

[0126] In this embodiment, an initial model can also be trained based on blind spot training data, window size annotation information, vehicle speed training data corresponding to the blind spot training data, and driving direction training data to obtain a loss value. Here, vehicle speed training data refers to the vehicle speed at which the camera captures the blind spot training data, and driving direction training data refers to the vehicle's driving direction at which the camera captures the blind spot training data. By jointly training the window recognition model with blind spot training data, window size annotation information, vehicle speed training data, and driving direction training data, the window recognition model can deeply learn the influence of information such as the correlation and continuity of obstacles between frames in the blind spot training data on the window size, as well as the influence of different vehicle speeds and driving directions on the window size. This allows it to learn the relationship between different blind spot images, vehicle speeds, driving directions, and window sizes, facilitating the subsequent adaptive determination of the display window size for displaying blind spot images based on this window recognition model, thus better assisting the driver in timely identification of driving risks.

[0127] In this embodiment, blind spot training data, vehicle speed training data, and driving direction training data can be input into an initial model to obtain the prediction window size. Then, the loss value is obtained using the prediction window size and window size annotation information. The initial model can be a neural network, specifically an attention-based neural network, a convolutional neural network, a fully connected neural network, or other types of neural networks. The specific form of the neural network can be determined according to the actual application scenario.

[0128] 902. Iteratively train the initial model based on the loss value to obtain the window recognition model.

[0129] In this embodiment, the parameters of the initial model can be adjusted based on the loss value, and the initial model can be iteratively trained.

[0130] After obtaining the window recognition model, it can be deployed on the vehicle system to enable direct real-time input of blind spot images for inference, and the model will output the window size corresponding to the blind spot image.

[0131] It should be noted that the specific expected inference results are directly related to the window size annotation information during the initial model pre-training. The breadth of blind zone training data and the accuracy of the window size annotation scale are necessary conditions for improving the correctness and rationality of the application use cases in this application. Model pre-training based on blind zone training data, due to information such as the correlation and continuity of obstacles between frames, plus the fact that vehicle speed training data and driving direction training data are also used as model input data, will also help improve the correctness and rationality of the application use cases in this application.

[0132] 903. Obtain blind spot images captured by the camera.

[0133] It is understood that 903 is the same as 701, 403, and 201 in the above embodiments, so it will not be described again.

[0134] 904. Input the blind spot image into the window recognition model to identify obstacle information around the vehicle and determine the window size.

[0135] It is understood that the parts of 904 that are the same as those of 202 in the above embodiment will not be described again here.

[0136] It should be noted that during the model training process, the window recognition model can learn the influence of information such as the correlation and continuity of obstacles between frames in the blind spot training data on the window size. This allows the window recognition model to identify the obstacle information around the vehicle corresponding to the blind spot image and output the corresponding window size when the blind spot image is input into the application.

[0137] The window recognition model can be trained using blind spot training data and the window size annotation information corresponding to the blind spot training data. Alternatively, the window recognition model can be trained using blind spot training data and the window size annotation information corresponding to the blind spot training data, vehicle speed training data, and driving direction training data.

[0138] In this embodiment, 904 may include: inputting blind spot images and vehicle information into a window recognition model to identify obstacle information around the vehicle and determine the window size. The vehicle information includes the vehicle's speed and direction of travel. Thus, in addition to the blind spot image, the vehicle's speed and direction of travel are combined to determine the window size, taking into account the influence of the vehicle's own motion state on the window size, enabling a more accurate and reasonable determination of the corresponding window size.

[0139] This application embodiment can, when a triggering condition is met, input the blind spot image into the window recognition model to identify obstacle information around the vehicle and determine the window size. The triggering condition can be at least one of the following: activating the vehicle's turn signal, the vehicle turning, or the vehicle changing lanes.

[0140] 905. Display the blind spot image on the screen using a display window of the corresponding window size.

[0141] It is understood that 905 is the same as 203 in the above embodiment, so it will not be described again.

[0142] As can be seen, in the method provided by the embodiments of this application, by pre-training the window recognition model, the window recognition model can learn the relationship between the blind spot image and the window size. Thus, after the blind spot image is input into the window recognition model, the size of the display window used to display the blind spot image can be adaptively determined, thereby better assisting the driver in timely identification of driving risks.

[0143] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0144] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0145] Please refer to Figure 10. An embodiment of this application provides a blind spot image display device 1000, which may include:

[0146] Module 1001 acquires blind spot images captured by the camera; the camera is mounted on the vehicle.

[0147] The determination module 1002 is used to determine the window size based on the obstacle information around the vehicle; the passage hazard level corresponding to the obstacle is positively correlated with the window size;

[0148] Display module 1003 is used to display blind spot images on the screen through a display window of a window size corresponding to the screen size.

[0149] As can be seen, in the method provided by this application, the traffic hazard level corresponding to the obstacle is positively correlated with the window size. Thus, based on the obstacle information around the vehicle, when the traffic hazard level is high, such as when the obstacle is close to the vehicle, a larger display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too small to see the obstacle clearly. When the traffic hazard level is low, such as when the obstacle is far from the vehicle, a smaller display window can be used to display the blind spot image on the screen, avoiding the problem that the existing fixed window is too large and affects the driver's use or viewing of navigation information. This achieves adaptive determination of the size of the display window used to display the blind spot image, so that the driver can better view the blind spot image through a display window of a reasonable size, thereby better assisting the driver in timely identification of driving risks.

[0150] In some embodiments of this application, the determining module 1002 in the blind spot image display device 1000 may include:

[0151] The determining unit is used to determine the distribution location information of obstacles in the blind spot image display area based on the obstacle information around the vehicle;

[0152] The unit is also used to determine the window size based on the distribution location information.

[0153] In some embodiments of this application, the determining unit is specifically used to determine the window size as a first set value if the obstacle is located in a first area of ​​the blind spot image display area; and to determine the window size as a second set value if the obstacle is located in a second area of ​​the blind spot image display area; wherein the passage danger level corresponding to the obstacle being located in the first area is greater than the passage danger level corresponding to the obstacle being located in the second area, and the first set value is greater than the second set value.

[0154] In some embodiments of this application, the determining unit in the blind spot image display device 1000 is specifically used to determine the relative relationship between the vehicle and the obstacle based on the blind spot image, vehicle information and obstacle information around the vehicle; and to determine the distribution location information based on the relative relationship.

[0155] In some embodiments of this application, the determining module 1002 in the blind spot image display device 1000 is further configured to determine a first area and a second area based on the performance of the vehicle; the traffic hazard level corresponding to the obstacle being located in the first area is greater than the traffic hazard level corresponding to the obstacle being located in the second area, and the distance between the first area and the vehicle is less than the distance between the second area and the vehicle.

[0156] The blind spot image display device 1000 also includes:

[0157] The construction module is used to construct the blind spot image display area based on the first and second regions.

[0158] In some embodiments of this application, the determining module 1002 in the blind spot image display device 1000 may include:

[0159] The determining unit is used to determine the relative relationship between the vehicle and the obstacles based on information about obstacles around the vehicle;

[0160] The unit is also used to determine the window size based on relative and mapping relationships; the mapping relationship is used to characterize the relationship between the relative relationship and the window size.

[0161] In some embodiments of this application, the relative relationships in the blind spot image display device 1000 include the relative distance between the vehicle and the obstacle; the mapping relationship includes: if the relative distance is greater than or equal to a first distance threshold, the window size is a third set value; if the relative distance is greater than a second distance threshold and less than the first distance threshold, the window size is a fourth set value; the fourth set value is greater than the third set value; if the relative distance is less than or equal to the second distance threshold, the window size is a fifth set value; the fifth set value is greater than the fourth set value.

[0162] In some embodiments of this application, the relative relationships in the blind spot image display device 1000 include: the relative distance between the vehicle and the obstacle, and the relative speed between the vehicle and the obstacle;

[0163] The defined unit includes:

[0164] Determine the sub-units to determine the weights corresponding to the relative velocities;

[0165] The updated sub-unit is used to update the relative distance based on the weights, thus obtaining the updated relative distance;

[0166] Determining sub-units is also used to determine the window size based on the updated relative distance and mapping relationship.

[0167] In some embodiments of this application, the determination module 1002 in the blind spot image display device 1000 is specifically used to input the blind spot image into the window recognition model to identify obstacle information around the vehicle and determine the window size; the window recognition model is trained by the blind spot training data and the window size annotation information corresponding to the blind spot training data.

[0168] In some embodiments of this application, the determination module 1002 in the blind spot image display device 1000 is specifically used to input the blind spot image and vehicle information into the window recognition model to identify obstacle information around the vehicle and determine the window size; the vehicle information includes: vehicle speed and driving direction.

[0169] In some embodiments of this application, the blind spot image display device 1000 further includes: a training module, used to train an initial model based on blind spot training data, window size annotation information, vehicle speed training data and driving direction training data corresponding to the blind spot training data, and obtain a loss value; and to iteratively train the initial model based on the loss value to obtain a window recognition model.

[0170] In some embodiments of this application, the determining module 1002 in the blind spot image display device 1000 is further used to determine the relative relationship between the vehicle and the obstacle based on the blind spot image, vehicle information and obstacle information around the vehicle.

[0171] The blind spot image display device 1000 also includes:

[0172] The adjustment module is used to adjust the viewing angle of blind spot images based on relative relationships.

[0173] In some embodiments of this application, the viewing angle in the blind spot image display device 1000 includes the field of view and the viewing focus;

[0174] The blind spot image display device 1000 also includes:

[0175] The cropping module is used to crop the blind zone image based on the field of view and the observation focus to obtain the cropped blind zone image;

[0176] Display module 1003 is specifically used to display the cropped blind spot image on the screen through a display window of a window size corresponding to the screen size.

[0177] In some embodiments of this application, the determining module 1002 in the blind spot image display device 1000 is specifically used to determine the window size based on the obstacle information around the vehicle when a trigger condition is met; the trigger condition is at least one of the following: the vehicle's turn signal is turned on, the vehicle is turning, or the vehicle is changing lanes.

[0178] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0179] This application also provides a computer storage medium, wherein the computer storage medium includes instructions that, when executed on a computer, cause the computer to perform some or all of the steps described in the above method embodiments.

[0180] The following describes a blind spot image display device provided in an embodiment of this application. Please refer to Figure 11, which is a schematic diagram of the structure of a blind spot image display device provided in an embodiment of this application. Specifically, the blind spot image display device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103, and a memory 1104 (the number of processors 1103 in the blind spot image display device 1000 can be one or more; Figure 11 shows one processor as an example). The processor 1103 may include an application processor 11031 and a communication processor 11032. In some embodiments of this application, the receiver 1101, transmitter 1102, processor 1103, and memory 1104 can be connected via a bus or other means.

[0181] Memory 1104 may include read-only memory and random access memory, and provides instructions and data to processor 1103. A portion of memory 1104 may also include non-volatile random access memory (NVRAM). Memory 1104 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0182] Processor 1103 controls the operation of the device. In specific applications, the various components of the device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0183] The blind spot image display method disclosed in the above embodiments of this application can be applied to, or implemented by, processor 1103. Processor 1103 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of processor 1103 or by instructions in software form. Processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1103 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1104. Processor 1103 reads the information in memory 1104 and, in conjunction with its hardware, completes the steps of the aforementioned blind spot image display method.

[0184] Receiver 1101 can be used to receive input digital or character information, and to generate signal inputs related to device settings and function control. Transmitter 1102 can be used to output digital or character information through the first interface; transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1102 may also include a display device such as a display screen.

[0185] In this embodiment, processor 1103 is used to execute the blind spot image display method in the embodiments corresponding to Figures 2, 4, 7, or 9. It should be noted that the specific manner in which the application processor 11031 in processor 1103 executes the aforementioned steps is based on the same concept as the method embodiments corresponding to Figures 1 to 9 in this application, and the resulting technical effects are the same as those in the method embodiments corresponding to Figures 1 to 9 in this application. For details, please refer to the descriptions in the method embodiments shown above in this application; further details will not be repeated here.

[0186] This application embodiment also provides a vehicle 1200. Please refer to Figure 12, which is a structural schematic diagram of a vehicle provided in this application embodiment. The vehicle 1200 is configured for fully or partially automated driving mode. For example, the vehicle 1200 can control itself while in automated driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of other vehicles performing possible behaviors, and control the vehicle 1200 based on the determined information. When the vehicle 1200 is in automated driving mode, the vehicle 1200 can also be set to operate without human interaction.

[0187] Vehicle 1200 may include various subsystems, such as a mobility system 1202, a sensor system 1204, a control system 1206, one or more peripheral devices 1208, a power supply 1210, a computer system 1212, and a user interface 1216. Optionally, vehicle 1200 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 1200 may be interconnected via wired or wireless means.

[0188] The mobility system 1202 may include components that provide powered motion to the vehicle 1200. In one embodiment, the mobility system 1202 may include an engine 1218, an energy source 1219, a transmission 1220, and wheels / tires 1221.

[0189] Engine 1218 can be an internal combustion engine, an electric motor, an air-compressed engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. Engine 1218 converts energy source 1219 into mechanical energy. Examples of energy source 1219 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 1219 can also provide energy to other systems of vehicle 1200. Transmission 1220 transmits mechanical power from engine 1218 to wheels 1221. Transmission 1220 may include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 1220 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 1221.

[0190] Sensor system 1204 may include several sensors for sensing information about the environment surrounding vehicle 1200. For example, sensor system 1204 may include a positioning system 1222 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 1224, a radar 1226, a laser rangefinder 1228, and a camera 1230. Sensor system 1204 may also include sensors for the internal systems of the monitored vehicle 1200 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of autonomous vehicle 1200.

[0191] The positioning system 1222 can be used to estimate the geographical location of the vehicle 1200. An IMU 1224 is used to sense changes in the position and orientation of the vehicle 1200 based on inertial acceleration. In one embodiment, the IMU 1224 can be a combination of an accelerometer and a gyroscope. A radar 1226 can use radio signals to sense objects in the surrounding environment of the vehicle 1200, specifically millimeter-wave radar or lidar. In some embodiments, in addition to sensing objects, the radar 1226 can also be used to sense the speed and / or direction of travel of objects. A laser rangefinder 1228 can use lasers to sense objects in the environment in which the vehicle 1200 is located. In some embodiments, the laser rangefinder 1228 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A camera 1230 can be used to capture multiple images of the surrounding environment of the vehicle 1200. The camera 1230 can be a still camera or a video camera.

[0192] The control system 1206 controls the operation of the vehicle 1200 and its components. The control system 1206 may include various components, including a steering system 1232, a throttle 1234, a braking unit 1236, a computer vision system 1240, a trajectory control system 1242, and an obstacle avoidance system 1244.

[0193] The steering system 1232 is operable to adjust the forward direction of the vehicle 1200. For example, in one embodiment, it may be a steering wheel system. The throttle 1234 controls the operating speed of the engine 1218 and thus the speed of the vehicle 1200. The braking unit 1236 controls the deceleration of the vehicle 1200. The braking unit 1236 may use friction to slow down the wheels 1221. In other embodiments, the braking unit 1236 may convert the kinetic energy of the wheels 1221 into electrical current. The braking unit 1236 may also take other forms to slow down the rotational speed of the wheels 1221 to control the speed of the vehicle 1200. The computer vision system 1240 is operable to process and analyze images captured by the camera 1230 to identify objects and / or features in the environment surrounding the vehicle 1200. Objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 1240 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 1240 can be used to map the environment, track objects, estimate object speeds, etc. The route control system 1242 is used to determine the driving route and speed of the vehicle 1200. In some embodiments, the route control system 1242 may include a lateral planning module 12421 and a longitudinal planning module 12422, which are respectively used to combine data from the obstacle avoidance system 1244, GPS 1222, and one or more predetermined maps to determine the driving route and speed for the vehicle 1200. The obstacle avoidance system 1244 is used to identify, evaluate, and avoid or otherwise traverse obstacles in the environment of the vehicle 1200, which may specifically be physical obstacles and virtual moving objects that may collide with the vehicle 1200. In one example, the control system 1206 may add or replace components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0194] Vehicle 1200 interacts with external sensors, other vehicles, other computer systems, or users via peripheral device 1208. Peripheral device 1208 may include wireless communication system 1246, on-board computer 1248, microphone 1250, and / or speaker 1252. In some embodiments, peripheral device 1208 provides a means for a user of vehicle 1200 to interact with user interface 1216. For example, on-board computer 1248 may provide information to a user of vehicle 1200. User interface 1216 may also operate on-board computer 1248 to receive user input. On-board computer 1248 may be operated via a touchscreen. In other cases, peripheral device 1208 may provide a means for vehicle 1200 to communicate with other devices located within the vehicle. For example, microphone 1250 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 1200. Similarly, speaker 1252 may output audio to a user of vehicle 1200. Wireless communication system 1246 may wirelessly communicate with one or more devices, either directly or via a communication network. For example, the wireless communication system 1246 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 1246 may utilize a wireless local area network (WLAN) for communication. In some embodiments, the wireless communication system 1246 may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system 1246 may include one or more dedicated short-range communications (DSRC) devices that may enable public and / or private data communication between the vehicle and / or a roadside station.

[0195] Power source 1210 can provide power to various components of vehicle 1200. In one embodiment, power source 1210 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 1200. In some embodiments, power source 1210 and energy source 1219 can be implemented together, as is the case in some fully electric vehicles.

[0196] Some or all of the functions of vehicle 1200 are controlled by computer system 1212. Computer system 1212 may include at least one processor 1213, which executes instructions 1215 stored in a non-transitory computer-readable medium such as memory 1214. Computer system 1212 may also be multiple computing devices that control individual components or subsystems of vehicle 1200 in a distributed manner. Processor 1213 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 1213 may be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although FIG12 functionally illustrates the processor, memory, and other components of computer system 1212 in the same block, those skilled in the art will understand that the processor or memory may actually include multiple processors or memories not stored in the same physical housing. For example, memory 1212 may be a hard disk drive or other storage media located in a housing different from that of computer system 1212. Therefore, references to processor 1213 or memory 1214 will be understood as references to a collection of processors or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as the steering assembly and deceleration assembly, may each have their own processor that performs only calculations relevant to the component's specific function.

[0197] In all aspects described herein, processor 1213 may be located remotely from vehicle 1200 and may communicate wirelessly with vehicle 1200. In other aspects, some of the processes described herein are executed on processor 1213 located within vehicle 1200 while others are executed by remote processor 1213, including taking the necessary steps to perform a single operation.

[0198] In some embodiments, memory 1212 may contain instructions 1215 (e.g., program logic) that can be executed by processor 1213 to perform various functions of vehicle 1200, including those described above. Memory 1214 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 1202, sensor system 1204, control system 1206, and peripheral devices 1208. In addition to instructions 1215, memory 1214 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 1200 and computer system 1212 during operation of vehicle 1200 in autonomous, semi-autonomous, and / or manual modes. User interface 1216 is used to provide or receive information from users of vehicle 1200. Optionally, the user interface 1216 may include one or more input / output devices within the set of peripheral devices 1208, such as a wireless communication system 1246, an in-vehicle computer 1248, a microphone 1250, and a speaker 1252.

[0199] Computer system 1212 can control the functions of vehicle 1200 based on inputs received from various subsystems (e.g., driving system 1202, sensor system 1204, and control system 1206) and from user interface 1216. For example, computer system 1212 can utilize inputs from control system 1206 to control steering system 1232 to avoid obstacles detected by sensor system 1204 and obstacle avoidance system 1244. In some embodiments, computer system 1212 is operable to provide control over many aspects of vehicle 1200 and its subsystems.

[0200] Alternatively, one or more of these components may be installed separately from or associated with the vehicle 1200. For example, the memory 1214 may exist partially or completely separately from the vehicle 1200. The components may be communicatively coupled together in a wired and / or wireless manner.

[0201] Optionally, the above components are merely examples. In practical applications, components in each of the above modules may be added or removed as needed. Figure 12 should not be construed as a limitation on the embodiments of this application. A vehicle traveling on a road, such as vehicle 1200 above, can identify objects in its surrounding environment to determine adjustments to its current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the vehicle can be determined.

[0202] Optionally, the vehicle 1200 or the computing devices associated with it, such as the computer system 1212, computer vision system 1240, and memory 1214 in Figure 12, can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of the others, so all identified objects can be considered together to predict the behavior of a single identified object. The vehicle 1200 can adjust its speed based on the predicted behavior of the identified objects. In other words, the vehicle 1200 can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of the vehicle 1200, such as the lateral position of the vehicle 1200 in the road, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing instructions to adjust the vehicle's speed, the computing device can also provide instructions to modify the steering angle of the vehicle 1200 so that the vehicle 1200 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the vehicle 1200 (e.g., cars in adjacent lanes on the road).

[0203] The aforementioned vehicle 1200 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, and train, etc., and this application embodiment does not impose any special limitations.

[0204] In this embodiment, the processor 1213 in the vehicle 1200 is used to execute the blind spot image display method in the embodiments corresponding to Figures 2, 4, 7, or 9. It should be noted that the specific manner in which the processor 1213 executes the aforementioned steps is based on the same concept as the method embodiments corresponding to Figures 1 to 9 in this application, and the resulting technical effects are the same as those in the method embodiments corresponding to Figures 1 to 9 in this application. For details, please refer to the descriptions in the aforementioned method embodiments of this application, which will not be repeated here.

[0205] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the blind spot image display method described in the embodiments shown in Figures 2, 4, 7, or 9.

[0206] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the blind spot image display method as described in the embodiments shown in Figures 2, 4, 7 or 9 above.

[0207] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the blind spot image display method described in the embodiments shown in Figures 2, 4, 7 or 9 above.

[0208] This application also provides a chip system including a processor for supporting the device in implementing the functions involved in the blind spot image display method described in the embodiments shown in Figures 2, 4, 7, or 9, such as sending or processing data and / or information involved in the blind spot image display method described in the embodiments shown in Figures 2, 4, 7, or 9. In one possible design, the chip system also includes a memory for storing necessary program instructions and data for the device. The chip system may be composed of chips or may include chips and other discrete devices.

[0209] This application also provides a chip, including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the blind spot image display method as described in the embodiments shown in FIG2, FIG4, FIG7 or FIG9 above.

[0210] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0211] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0213] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0214] The aforementioned computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for displaying images in blind spots, characterized in that, The method includes: Acquire blind spot images captured by a camera; the camera is mounted on the vehicle; The window size is determined based on information about obstacles around the vehicle; the traffic hazard level corresponding to the obstacle is positively correlated with the window size. The blind spot image is displayed on the screen through a display window of the specified window size.

2. The method according to claim 1, characterized in that, Determining the window size includes: Based on the obstacle information around the vehicle, determine the distribution location information of the obstacles in the blind spot image display area; The window size is determined based on the distribution location information.

3. The method according to claim 2, characterized in that, Determining the window size based on the distribution location information includes: If the obstacle is located in the first area of ​​the blind spot image display area, the window size is determined to be a first set value; If the obstacle is located in the second region of the blind spot image display area, the window size is determined to be a second set value; Wherein, the passage hazard level corresponding to the obstacle being located in the first area is greater than the passage hazard level corresponding to the obstacle being located in the second area, and the first set value is greater than the second set value.

4. The method according to any one of claims 2-3, characterized in that, Determining the distribution location information includes: Based on the blind spot image, vehicle information, and obstacle information around the vehicle, the relative relationship between the vehicle and the obstacle is determined; The distribution location information is determined based on the relative relationship.

5. The method according to any one of claims 2-4, characterized in that, The method further includes: Based on the vehicle's performance, a first area and a second area are determined; the traffic hazard level corresponding to the obstacle located in the first area is greater than the traffic hazard level corresponding to the obstacle located in the second area, and the distance between the first area and the vehicle is less than the distance between the second area and the vehicle; Based on the first region and the second region, the blind spot image display area is constructed.

6. The method according to claim 1, characterized in that, Determining the window size includes: Based on the obstacle information around the vehicle, the relative relationship between the vehicle and the obstacle is determined; The window size is determined based on the relative relationship and the mapping relationship; the mapping relationship is used to characterize the relationship between the relative relationship and the window size.

7. The method according to claim 6, characterized in that, The relative relationship includes the relative distance between the vehicle and the obstacle; The mapping relationship includes: If the relative distance is greater than or equal to the first distance threshold, the window size is a third preset value; If the relative distance is greater than the second distance threshold and less than the first distance threshold, the window size is a fourth set value; the fourth set value is greater than the third set value. If the relative distance is less than or equal to the second distance threshold, the window size is a fifth set value; the fifth set value is greater than the fourth set value.

8. The method according to any one of claims 6-7, characterized in that, The relative relationship includes: the relative distance between the vehicle and the obstacle, and the relative speed between the vehicle and the obstacle; Determining the window size based on the relative and mapping relationships includes: Determine the weights corresponding to the relative velocities; The relative distance is updated based on the weights to obtain the updated relative distance; The window size is determined based on the updated relative distance and the mapping relationship.

9. The method according to claim 1, characterized in that, Determining the window size includes: The blind spot image input window recognition model identifies obstacle information around the vehicle and determines the window size; the window recognition model is trained using blind spot training data and the window size annotation information corresponding to the blind spot training data.

10. The method according to claim 9, characterized in that, The step of identifying obstacle information around the vehicle using the blind spot image input window recognition model and determining the window size includes: The blind spot image and vehicle information are input into the window recognition model to identify obstacle information around the vehicle and determine the window size; the vehicle information includes the vehicle speed and driving direction.

11. The method according to any one of claims 9-10, characterized in that, The method further includes: The initial model is trained based on the blind spot training data, the window size annotation information, the vehicle speed training data and driving direction training data corresponding to the blind spot training data, and the loss value is obtained. The initial model is iteratively trained based on the loss value to obtain the window recognition model.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: Based on the blind spot image, vehicle information, and obstacle information around the vehicle, the relative relationship between the vehicle and the obstacle is determined; Based on the aforementioned relative relationship, the viewing angle of the blind spot image is adjusted.

13. The method according to claim 12, characterized in that, The observation angle includes the field of view and the observation focus, and the method further includes: The blind zone image is cropped according to the field of view and the observation focus to obtain the cropped blind zone image; The display of the blind spot image includes: The cropped blind spot image is displayed on the screen through a display window corresponding to the window size.

14. The method according to any one of claims 1-13, characterized in that, Determining the window size includes: When the triggering conditions are met, the window size is determined based on the obstacle information around the vehicle; the triggering conditions are at least one of the following: the vehicle's turn signal is turned on, the vehicle is turning, or the vehicle is changing lanes.

15. A blind spot image display device, characterized in that, The device includes: The acquisition module acquires blind spot images captured by the camera; the camera is mounted on the vehicle. The determination module is used to determine the window size based on information about obstacles around the vehicle; the passage hazard level corresponding to the obstacle is positively correlated with the window size; The display module is used to display the blind spot image on the screen through a display window of the specified window size.

16. A blind spot image display device, comprising a processor and a memory, the memory and the processor being coupled, the processor being configured to execute the blind spot image display method according to any one of claims 1 to 14.

17. A computer-readable storage medium comprising instructions, when executed on a computer, causing the computer to perform the blind spot image display method according to any one of claims 1 to 14.

18. A computer program product containing instructions that, when run on a computer, causes the computer to perform the blind spot image display method as described in any one of claims 1 to 14.

19. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the blind spot image display method according to any one of claims 1 to 14.

20. A vehicle, characterized in that, It includes a processor and a memory, the memory and the processor being coupled together, the processor being used to execute the blind spot image display method according to any one of claims 1 to 14.

Images