Door opening early warning method and device, computer program product and electronic equipment

Abstract

The invention belongs to the technical field of vehicle safety, and particularly relates to a door opening early warning method and device, a computer program product and electronic equipment. The method comprises the following steps: respectively determining a first distance measurement result and a second distance measurement result under the condition that vehicle basic information meets a door opening early warning trigger condition; determining a third distance measurement result based on the first weight, the second weight, the first distance measurement result and the second distance measurement result; the first weight is the weight corresponding to the first ranging result, and the second weight is the weight corresponding to the second ranging result; performing obstacle type detection on each to-be-detected image based on the third distance measurement result to obtain an obstacle type detection result; and performing door opening early warning based on the obstacle type detection result.

Description

Technical Field

[0001] The present application belongs to the field of vehicle safety technology, and in particular, relates to a door opening warning method, device, computer program product and electronic equipment. Background Art

[0002] Vehicle door opening warning is an important innovation in the field of vehicle safety technology. It can effectively reduce traffic accidents caused by careless door opening by monitoring the vehicle's environment in real time and actively warning of potential risks. However, existing vehicle door opening warning solutions usually use radar to detect the vehicle environment, but the radar's detection accuracy and detection distance are limited, making it difficult to accurately warn of vehicle door opening, resulting in a high safety hazard of vehicle door opening. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a door opening warning method, device, computer program product and electronic device to solve the problem that the door opening warning method in the prior art has low accuracy, resulting in a high safety hazard when opening the vehicle door.

[0004] A first aspect of an embodiment of the present application provides a door opening warning method, which may include:

[0005] When the basic information of the vehicle meets the door opening warning triggering condition, a first ranging result and a second ranging result are determined respectively; the first ranging result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second ranging result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device;

[0006] Determine a third ranging result based on a first weight, a second weight, the first ranging result, and the second ranging result; the first weight is a weight corresponding to the first ranging result, and the second weight is a weight corresponding to the second ranging result;

[0007] Performing obstacle type detection on each image to be detected based on the third distance measurement result to obtain an obstacle type detection result; any of the images to be detected is an image captured by any of the camera acquisition devices;

[0008] A door opening warning is performed based on the obstacle type detection result.

[0009] A second aspect of an embodiment of the present application provides a door opening warning device, which may include:

[0010] A first determination module is used to determine a first distance measurement result and a second distance measurement result respectively when the basic information of the vehicle meets the door opening warning triggering condition; the first distance measurement result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second distance measurement result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device;

[0011] A second determination module, configured to determine a third ranging result based on a first weight, a second weight, the first ranging result, and the second ranging result; the first weight is a weight corresponding to the first ranging result, and the second weight is a weight corresponding to the second ranging result;

[0012] A detection module, configured to perform obstacle type detection on each image to be detected based on the third distance measurement result to obtain an obstacle type detection result; any of the images to be detected is an image captured by any of the camera acquisition devices;

[0013] The early warning module is used to issue a door opening early warning based on the obstacle type detection result.

[0014] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned door opening warning methods are implemented.

[0015] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the steps of any one of the above-mentioned door opening warning methods.

[0016] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is run, any of the above-mentioned door opening warning methods is executed.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: when the basic information of the vehicle meets the triggering condition of the door opening warning, the embodiments of the present application respectively determine the first distance measurement result and the second distance measurement result; the first distance measurement result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second distance measurement result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device; based on the first weight, the second weight, the first distance measurement result and the second distance measurement result, determine the third distance measurement result; the first weight is the weight corresponding to the first distance measurement result, and the second weight is the weight corresponding to the second distance measurement result; based on the third distance measurement result, each image to be detected is subjected to obstacle type detection to obtain an obstacle type detection result; and a door opening warning is performed based on the obstacle type detection result. Through the embodiments of the present application, the distance measurement result (first distance measurement result) determined by the ultrasonic radar and the distance measurement result (second distance measurement result) determined by the camera acquisition device can be combined, which can greatly improve the detection accuracy and detection distance in complex environments, accurately perform vehicle door opening warning, and ensure the safety of vehicle door opening. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 Schematic diagram of the sensing range of the camera acquisition device in the embodiment of the present application;

[0020] Figure 2 A schematic diagram of the sensing range of the ultrasonic radar in an embodiment of the present application;

[0021] Figure 3 This is a flow chart of an embodiment of a door opening warning method in an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of displaying information in an embodiment of the present application;

[0023] Figure 5 This is another schematic diagram of displaying information in an embodiment of the present application;

[0024] Figure 6 A schematic diagram showing a viewing angle in an embodiment of the present application;

[0025] Figure 7This is a structural diagram of an embodiment of a door opening warning device in an embodiment of the present application;

[0026] Figure 8 This is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Vehicle door opening warning is an important innovation in the field of vehicle safety technology. It can effectively reduce traffic accidents caused by careless door opening by monitoring the vehicle's environment in real time and actively warning of potential risks. However, existing vehicle door opening warning solutions usually use radar to detect the vehicle environment, but the radar's detection accuracy and detection distance are limited, making it difficult to accurately warn of vehicle door opening, resulting in a high safety hazard of vehicle door opening.

[0034] In view of this, the embodiments of the present application provide a door opening warning method, device, computer program product and electronic device to solve the problem that the door opening warning method in the prior art has low accuracy, resulting in a high safety hazard when the vehicle door is opened.

[0035] It should be noted that the executor of the method of the present application is an electronic device, which may be a vehicle, an on-board terminal deployed on the vehicle, or a server.

[0036] In the embodiment of the present application, at least four video acquisition devices (such as cameras) can be set on the vehicle to cover the front, rear, left and right sides of the vehicle respectively. Figure 1 As shown, a panoramic image of the vehicle's environment is collected, and environmental perception can be performed based on the collected image.

[0037] In the embodiment of the present application, one or more ultrasonic radars may be arranged on the vehicle. Specifically, an ultrasonic radar may be arranged on the rear side of the vehicle. Figure 2 shown.

[0038] Optionally, an ultrasonic radar may be disposed on the front side of the vehicle; the ultrasonic radar may send ultrasonic transmission signals and may perform ranging based on received ultrasonic echo signals to achieve environmental perception of the front and / or rear side of the vehicle.

[0039] In addition, the embodiment of the present application may further arrange a side-view imaging acquisition device (such as a side-view camera) on the left and right sides of the vehicle respectively to capture images of the side and rear of the vehicle, and may perform environmental perception of the side and rear of the vehicle based on the captured images.

[0040] In an embodiment of the present application, it is possible to detect whether the user has an intention to park, and when it is detected that the user has an intention to park, a door opening warning can be issued in a timely manner; specifically, basic vehicle information can be obtained, and based on the basic vehicle information, it can be determined whether the user has an intention to park; if the basic vehicle information meets the preset warning trigger conditions, it can be considered that the user has an intention to park; if the basic vehicle information does not meet the preset warning trigger conditions, it can be considered that the user has no intention to park.

[0041] Here, the basic vehicle information may include the vehicle speed, and accordingly, the door opening warning trigger condition may be that the vehicle speed is less than or equal to a preset vehicle speed threshold; or, the basic vehicle information may include the gear position, and accordingly, the door opening warning trigger condition may be that the vehicle gear position is the parking gear (P gear).

[0042] The vehicle speed threshold can be set specifically and situationally according to actual needs, and the embodiment of the present application does not limit this. For example, the vehicle speed threshold can be set to 5 kilometers per hour or 3 kilometers per hour.

[0043] In an embodiment of the present application, if the vehicle speed is less than or equal to a preset speed threshold, or the vehicle gear is in the parking gear (P gear), it can be considered that the basic vehicle information meets the door opening warning triggering conditions. At this time, it can be considered that the user has the intention to park.

[0044] When the user intends to park, the vehicle door opening warning process needs to be carried out in a timely manner to better ensure the user's safety. Figure 3 In an embodiment of the present application, an embodiment of a door opening warning method may include steps S301 to S304:

[0045] Step S301: When the basic information of the vehicle meets the door opening warning triggering condition, a first distance measurement result and a second distance measurement result are determined respectively.

[0046] Among them, the first ranging result is used to reflect the first position information and / or first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second ranging result is used to reflect the second position information and / or second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device.

[0047] Here, the smart device may be a vehicle, and the first ranging result may be used to reflect the first position information and / or first distance of one or more first obstacles measured by one or more ultrasonic radars deployed on the vehicle, and the second ranging result may be used to reflect the second position information and / or second distance of each second obstacle measured by multiple camera acquisition devices deployed on the vehicle.

[0048] Specifically, if the vehicle is equipped with one ultrasonic radar (for example, one ultrasonic radar is equipped on the rear side of the vehicle), the first ranging result can be determined based on the one ultrasonic radar; if the vehicle is equipped with multiple ultrasonic radars (for example, one ultrasonic radar is equipped on the front and rear sides of the vehicle), the first ranging result can be determined based on multiple ultrasonic radars.

[0049] In an embodiment of the present application, one or more ultrasonic radars can be used to send ultrasonic transmission signals to detect obstacles near the vehicle; when the ultrasonic transmission signal encounters an obstacle, part of the ultrasonic transmission signal will be reflected, and the ultrasonic radar can receive the reflected ultrasonic reflection signal (also called ultrasonic echo signal).

[0050] Afterwards, the distance measurement information of each first obstacle can be determined based on each ultrasonic echo signal; here, the obstacle determined based on the ultrasonic radar can be referred to as the first obstacle.

[0051] Specifically, after receiving the ultrasonic echo signal, the reception time of the ultrasonic echo signal and the transmission time of the ultrasonic transmit signal corresponding to the ultrasonic echo signal can be determined; then, based on the time difference between the reception time and the transmission time, the distance between the first obstacle corresponding to the ultrasonic echo signal and the vehicle can be calculated; in addition, based on the direction corresponding to the ultrasonic echo signal and the distance, the position information of the first obstacle (for example, expressed in coordinates) can be determined.

[0052] Here, the distance between the first obstacle and the vehicle may be referred to as the first distance, and the position information of the first obstacle may be referred to as the first position information; for each first obstacle, the first distance and / or the first position information respectively corresponding to the first obstacle may be determined as the ranging information of the first obstacle; accordingly, the ranging information of each first obstacle may be obtained.

[0053] Afterwards, the distance measurement information of each first obstacle may be determined as the first distance measurement result.

[0054] In the embodiment of the present application, multiple camera acquisition devices can also be used to determine the second distance measurement result. Specifically, each camera acquisition device can be used to acquire each frame of image (referred to as the image to be detected); here, for each camera acquisition device, adjacent frames of images to be detected can be acquired; then, two or more adjacent frames of images to be detected acquired by the same camera acquisition device can be used to determine obstacles and measure the distance of obstacles.

[0055] Specifically, two or more adjacent frames of images to be detected can be compared to determine the degree of change of pixels at various positions in the two or more adjacent frames of images to be detected; if the degree of change of pixels in a certain area in the two or more adjacent frames of images to be detected is large, it can be considered that the area includes obstacles; if the degree of change of pixels in a certain area in the two or more adjacent frames of images to be detected is small, it can be considered that the area does not include obstacles. Based on this, various obstacles can be determined.

[0056] Here, the obstacle determined based on the camera acquisition device may be referred to as a second obstacle, the position information of the second obstacle may be referred to as second position information (for example, may be represented by coordinates), and the distance of the second obstacle may be referred to as a second distance.

[0057] In the embodiment of the present application, based on the area where each second obstacle is located, the second position information of the second obstacle can be determined. As an example, the coordinates of the center of the area where the second obstacle is located can be determined as the second position information of the second obstacle.

[0058] In addition, the embodiment of the present application can also pre-calibrate multiple camera acquisition devices to determine the parameters (including internal parameters and external parameters) corresponding to the multiple camera acquisition devices respectively; based on this, the depth information of each pixel point in the image to be detected captured by the camera acquisition device can be determined.

[0059] Based on the depth information of the second position information of the second obstacle, a second distance between the second obstacle and the vehicle may be determined.

[0060] For each second obstacle, the second position information and / or the second distance of the second obstacle can be determined as the ranging information of the second obstacle; accordingly, the ranging information of each second obstacle can be obtained; then, the ranging information of each second obstacle can be determined as the second ranging result.

[0061] It should be understood that the embodiments of the present application do not limit the execution order of the above-mentioned first ranging result determination process and the second ranging result determination process, and the execution order can be determined according to actual needs. For example, the first ranging result determination process can be executed first, and then the second ranging result determination process can be executed. For another example, the second ranging result determination process can be executed first, and then the first ranging result determination process can be executed. For another example, the first ranging result determination process and the second ranging result determination process can be executed in parallel.

[0062] Step S302: Determine a third ranging result based on the first weight, the second weight, the first ranging result, and the second ranging result.

[0063] Since ultrasonic radar has limited detection accuracy and a short detection distance (usually 0.3 meters to 5 meters), it is difficult to obtain accurate detection results when detecting static objects (such as signal traffic sign poles, fire hydrants, curbs, roadblocks, isolation piers, trees, etc.) and fast-moving dynamic objects; in addition, due to cost constraints, most vehicles may only have ultrasonic radars on the front and rear sides, so it is impossible to identify obstacles on the left and right sides of the vehicle; in order to improve the accuracy and range of environmental perception, the embodiments of the present application can integrate the perception results of the ultrasonic radar and the perception results of the camera acquisition device to perceive and monitor the vehicle environment in multiple dimensions.

[0064] In the embodiment of the present application, for an obstacle detected only in the first ranging result, the ranging information of the obstacle in the first ranging result can be directly determined as the finally obtained ranging information; for an obstacle detected only in the second ranging result, the ranging information of the obstacle in the second ranging result can be directly determined as the finally obtained ranging information; for an obstacle detected in both the first ranging result and the second ranging result, the finally obtained ranging information can be determined based on the ranging information of the obstacle in the first ranging result and the ranging information in the second ranging result.

[0065] Specifically, the intersection of each first obstacle and each second obstacle can be obtained, and each obstacle in the intersection can be determined as each third obstacle; that is, the third obstacle is an obstacle detected in both the first ranging result and the second ranging result. In the embodiment of the present application, if the distance between the coordinates (first position information) of a first obstacle and the coordinates (second position information) of a second obstacle is less than a preset distance threshold, it can be considered that the first obstacle and the second obstacle are actually the same obstacle, and the obstacle can be determined as the third obstacle.

[0066] In addition, a difference set between each first obstacle and each second obstacle can also be obtained, and the difference set includes each obstacle that belongs to the first obstacle but does not belong to the second obstacle; then, each obstacle in the difference set can be determined as each first specific obstacle; that is, the first specific obstacle is an obstacle that is only detected in the first ranging result. In the embodiment of the present application, if the distance between the coordinates (first position information) of a first obstacle and the coordinates (second position information) of each second obstacle is greater than or equal to a preset distance threshold, then the first obstacle can be considered to be an obstacle that is only detected in the first ranging result, and the obstacle can be determined as the first specific obstacle.

[0067] Similarly, the difference set between each second obstacle and each first obstacle can also be obtained, and the difference set includes each obstacle that belongs to the second obstacle but not to the first obstacle; then, each obstacle in the difference set can be determined as a second unique obstacle; that is, the second unique obstacle is an obstacle that is only detected in the second ranging result. In the embodiment of the present application, if the distance between the coordinates (second position information) of a second obstacle and the coordinates (first position information) of each first obstacle is greater than or equal to a preset distance threshold, then the second obstacle can be considered to be an obstacle that is only detected in the second ranging result, and the obstacle can be determined as a second unique obstacle.

[0068] The above distance threshold can be specified and set according to actual needs, and the embodiments of the present application are not limited to this.

[0069] In the embodiment of the present application, for each first unique obstacle, the ranging information of the corresponding first obstacle can be directly determined as the final ranging result of the first unique obstacle; similarly, for each second unique obstacle, the ranging information of the corresponding second obstacle can be directly determined as the final ranging result of the second unique obstacle; and for each third obstacle, it is necessary to comprehensively consider the ranging information of the corresponding first obstacle and the second obstacle to determine a more robust and accurate ranging result.

[0070] Specifically, a corresponding first weight may be set for the first ranging result, and a corresponding second weight may be set for the second ranging result. After that, the two ranging results may be fused using the first weight and the second weight. Here, based on the first weight and the second weight, the ranging information of the first obstacle and the ranging information of the second obstacle corresponding to each third obstacle may be weighted averaged to obtain the ranging information of each third obstacle. For example, the ranging information of obstacle A in the first ranging result is ranging information 1, and the ranging information of obstacle A in the second ranging result is ranging information 2. Then, the first weight and the second weight may be used to weight average ranging information 1 and ranging information 2 to obtain the corresponding ranging information 3.

[0071] The specific values ​​of the first weight and the second weight can be specific and situational according to actual needs, and the embodiments of the present application do not limit this. For example, it can be considered that the ranging result of the ultrasonic radar is more accurate, so the first weight can be set to a larger value, and accordingly, the second weight can be set to a smaller value. For another example, the ranging result of the ultrasonic radar and the ranging result of the camera acquisition device can be considered equally, so that the first weight and the second weight can be set to the same value.

[0072] Specifically, based on the first weight and the second weight, the first position information and the second position information corresponding to the third obstacle can be weighted averaged to obtain the corresponding third position information; based on the third position information, the third distance of the third obstacle can be determined. Based on the third position information and / or the third distance of the third obstacle, the final distance measurement information corresponding to the third obstacle can be determined. Accordingly, the distance measurement information of each third obstacle can be obtained.

[0073] Afterwards, the ranging information of each first unique obstacle, the ranging information of each second unique obstacle, and the ranging information of each third obstacle may be determined as a third ranging result.

[0074] Step S303: performing obstacle type detection on each to-be-detected image based on the third distance measurement result to obtain an obstacle type detection result.

[0075] In the embodiment of the present application, after determining the distance of each obstacle, the obstacle type corresponding to each detected obstacle can also be identified, so that more abundant obstacle information can be obtained to facilitate subsequent obstacle display and prompting.

[0076] Here, the detected obstacle may be referred to as a fourth obstacle, that is, each of the first specific obstacles, each of the second specific obstacles, and each of the third obstacles may be determined as a fourth obstacle.

[0077] In the embodiment of the present application, the images to be detected collected by each camera acquisition device can be used to detect the obstacle type. Specifically, based on the ranging information of each fourth obstacle, the area where each fourth obstacle is located in the image to be detected (referred to as the area to be detected) can be determined, and then the obstacle type detection can be performed on the area to be detected. Based on this, the specific area where the obstacle type detection needs to be performed can be determined, which can greatly improve the detection efficiency.

[0078] For example, based on the ranging information of the fourth obstacle A, it can be determined that the fourth obstacle A is located at the rear side of the vehicle, and then the obstacle type detection of obstacle A can be performed based on the image 1 to be detected acquired by the camera acquisition device located at the rear side of the vehicle; specifically, the area in the image 1 to be detected where the coordinate 2 of the fourth obstacle A is located can be determined, where the coordinate 2 is in area 3 of the image 1 to be detected, and then area 3 can be determined as the area to be detected corresponding to the fourth obstacle A; thereafter, the obstacle type detection of the fourth obstacle A can be performed on area 3.

[0079] In the embodiment of the present application, a preset target detection model can be used to identify the obstacle type of each fourth obstacle. The target detection model can be any artificial intelligence model for target detection, including but not limited to YOLO series, region-based convolutional neural network (R-CNN), fast region-based convolutional network (Fast Region-based Convolutional Network, Fast R-CNN), region proposal network (RPN) and other artificial intelligence models for target detection, which are not limited in the embodiment of the present application.

[0080] Here, obstacle types may include various common objects in driving scenarios, including but not limited to pedestrians, bicycles, electric vehicles, tricycles, vehicles, signal traffic sign poles, fire hydrants, curbs, roadblocks, isolation piers, trees and other objects.

[0081] In a specific implementation of the embodiment of the present application, when computing resources are limited, the type of obstacle can be preliminarily determined. The types of obstacles referred to here can include dynamic objects such as pedestrians, bicycles, electric vehicles, tricycles, and vehicles, as well as static objects such as signal traffic sign poles, fire hydrants, curbs, roadblocks, isolation piers, and trees; if the obstacle is preliminarily determined to be a dynamic object, it can be considered that the obstacle is highly dangerous, and the target detection model can be used to further determine the specific type of the obstacle; if the obstacle is preliminarily determined to be a static object, it can be considered that the obstacle is less dangerous, and when computing resources are limited, there is no need to use the target detection model to further identify the specific type of the obstacle. Specifically, the degree of change of each pixel corresponding to the position of the obstacle in two or more adjacent frames of the image to be detected can be determined. If the degree of change is large, it can be preliminarily determined that the obstacle is a dynamic object, and the target detection model can be used to further determine the specific type of the obstacle; if the degree of change is small, it can be preliminarily determined that the obstacle is a static object, and at this time, there is no need to use the target detection model to further identify the specific type of the obstacle, so as to save computing resources.

[0082] In another specific implementation of the embodiment of the present application, it is considered that shadows may affect the accuracy of obstacle recognition. For example, shadows may cause the model to mistakenly identify shadows as obstacles, or identify shadows as part of obstacles, making it difficult for the target detection algorithm to correctly identify the type of obstacles. Therefore, here, the obstacles can be shadowed accordingly to improve the detection accuracy; specifically, if the obstacle is identified as a static object, the obstacle can be shadowed; for example, a median filter is used to smooth the shadow edge, or a morphological operation is used to remove small shadow areas or fill small holes in the shadow area to reduce the impact of shadows on obstacle type recognition.

[0083] In another specific implementation of the embodiment of the present application, after obstacle type detection is performed, the detection result may be post-processed to further optimize and correct the detection result to improve the accuracy and robustness of the detection.

[0084] As an example, the post-processing process may include, but is not limited to, one or more processes such as non-maximum suppression, filter smoothing, geometric correction, and scale filtering.

[0085] In the embodiment of the present application, the obstacle type, distance (referred to as the fourth distance), and position information (referred to as the fourth position information) of each fourth obstacle may be determined as the obstacle type detection result.

[0086] Step S304: Perform door opening warning based on the obstacle type detection result.

[0087] In the embodiment of the present application, in order to improve the safety of users getting off the vehicle, it is necessary to perform a door opening warning based on the obstacle type detection result to better remind the user.

[0088] Specifically, the user's getting-off position can be determined, and then a door opening warning can be given in a targeted manner based on the user's getting-off position to better meet user needs in actual application scenarios.

[0089] Here, the basic vehicle information may also include a seat belt status and / or a door opening status; based on the basic vehicle information, the user's getting-off direction may be determined.

[0090] For example, the basic vehicle information includes the seat belt status. Based on the seat belt status, it can be determined that the seat belt of the main driver's seat of the vehicle is unfastened, so the direction corresponding to the main driver's seat of the vehicle can be determined as the direction of getting off. For another example, the basic vehicle information includes the seat belt status. Based on the seat belt status, it can be determined that the seat belt of the co-driver's seat and the left side of the rear seat of the vehicle are unfastened, so the direction corresponding to the co-driver's seat and the left side of the rear seat of the vehicle can be determined as the direction of getting off.

[0091] For another example, the basic vehicle information may include a door opening status. Based on the door opening status, it can be determined that the left rear door of the vehicle is opened, and therefore the direction corresponding to the left rear door of the vehicle can be determined as the getting-off direction.

[0092] For another example, basic vehicle information may include seat belt status and door opening status. Based on the seat belt status, it can be determined that the seat belt of the vehicle's main driver's seat is unfastened; based on the door opening status, it can be determined that the vehicle's left rear door is opened. Therefore, the position corresponding to the vehicle's main driver's seat and the position corresponding to the vehicle's left rear door can be determined as the getting off position.

[0093] After determining one or more exit directions, a door opening warning can be provided based on the obstacle type detection result to remind the user of the situation surrounding the vehicle; specifically, the door opening warning in the embodiment of the present application may include obstacle information display and / or voice prompt.

[0094] The following will introduce in detail the process of obstacle information display and voice prompt process.

[0095] In the embodiment of the present application, corresponding display information can be generated based on the obstacle type detection result. Specifically, the obstacle type detection result can include the fourth distance, fourth position information and obstacle type corresponding to each fourth obstacle; for each fourth obstacle, the fourth distance, fourth position information and display information corresponding to the obstacle type corresponding to the fourth obstacle can be determined.

[0096] As an example, based on the fourth distance of the fourth obstacle, the corresponding distance display text on the image to be detected can be determined; based on the obstacle type of the fourth obstacle, the corresponding target type display text on the image to be detected can be determined; in addition, based on the fourth position information of the fourth obstacle, the corresponding annotation box of the fourth obstacle on the image to be detected can be determined, and the image to be detected, distance display text, target type display text, and annotation box corresponding to the fourth obstacle can be used as the display information corresponding to the fourth obstacle.

[0097] Afterwards, various display information can be displayed on one or more preset display screens. The display screen referred to in the embodiment of the present application can be specifically arranged on the dashboard and the rear row of the vehicle.

[0098] For example, if the fourth distance of the fourth obstacle A is 30 centimeters (cm), and the obstacle type of the fourth obstacle is a pedestrian, the corresponding distance display text "30cm" can be determined based on 30 centimeters, and the corresponding target type display text "pedestrian" or "pedestrian approaching" can also be determined based on the pedestrian (obstacle type); in addition, the label box corresponding to the fourth obstacle A can also be determined based on the fourth position information of the fourth obstacle A, and the above display information can be displayed on one or more display screens, such as Figure 4 shown.

[0099] For another example, if the fourth distance of the fourth obstacle B is 30 centimeters (cm), and the obstacle type of the fourth obstacle is a bicycle, the corresponding distance display text "30cm" can be determined based on 30 centimeters, and the corresponding target type display text "bicycle" or "bicycle approaching" can also be determined based on the bicycle; in addition, based on the fourth position information of the fourth obstacle B, a label box corresponding to the fourth obstacle B can be determined, and the above display information can be displayed on one or more display screens, such as Figure 5 shown.

[0100] In the embodiment of the present application, a display screen corresponding to the getting-off direction (referred to as a target display screen) can also be determined. For example, if the getting-off direction is the direction corresponding to the main driving position, the display screen located at the main driving position can be determined as the target display screen. In addition, display information corresponding to the getting-off direction (referred to as target display information) can also be determined. For example, if the getting-off direction is the direction where the main driving position is located, the display information of the fourth obstacle whose fourth position information is located near the main driving position can be determined as the target display information. Thereafter, the target display information can be played using the target display screen, thereby reducing the visual interference of irrelevant environmental information to the user, allowing the user to more quickly and accurately obtain the environmental conditions of his or her own location.

[0101] In a specific implementation of the embodiment of the present application, the specific display mode can be customized by the user; for example, the user can choose to display various display information on multiple display screens simultaneously, for example, displaying the display information of various directions such as the left side, right side, front side, and rear side of the vehicle on various display screens at the same time; for another example, the user can choose to display the display information of corresponding directions on different display screens, for example, the user can choose to display only the display information corresponding to the left side and the rear side of the vehicle on the display screen on the left rear row of the vehicle, and only display the display information corresponding to the right side and the rear side of the vehicle on the display screen on the right rear row of the vehicle, and only display the display information corresponding to the left side and the right side of the vehicle on the display screen on the instrument panel; for another example, the user can choose to display the display information obtained by a specific camera acquisition device, for example, the user can choose to display only the display information obtained by the camera acquisition devices arranged on the left and right sides and the two side view camera acquisition devices, such as Figure 6 As shown, only the left and right perspectives and the side and rear perspectives can be displayed, so that the display can be more targeted.

[0102] By displaying information about each obstacle on the display screen, the vehicle's surrounding environment can be displayed more richly, allowing users to more easily understand the environment around the vehicle, especially in conditions of poor visibility, such as rainy and snowy weather, parking at night, etc., and can help users identify potential dangers more quickly, thereby reducing the occurrence of accidents.

[0103] In the embodiment of the present application, corresponding voice prompt information can also be determined based on the obstacle type detection result.

[0104] Specifically, for each fourth obstacle, corresponding voice prompt information may be determined based on the fourth distance, fourth position information, and obstacle type corresponding to the fourth obstacle.

[0105] Afterwards, one or more sound units may be used to play the prompt voices corresponding to the various voice prompt messages. The sound units referred to herein may be speakers or amplifiers located at different positions of the vehicle (e.g., the main driver's seat, the co-driver's seat, the left and right sides of the rear row of the vehicle, etc.).

[0106] As an example, the corresponding voice prompt information can be determined based on the fourth distance, fourth position information, and type of the fourth obstacle; for example, the fourth distance of the fourth obstacle B is 30 centimeters (cm), the obstacle type of the fourth obstacle B is a bicycle, and the fourth position information of the fourth obstacle indicates that the fourth obstacle is located on the right side of the rear seat of the vehicle. Based on the above information, the voice prompt information "A bicycle is approaching from the right rear of the vehicle, 30 centimeters away" can be determined, and the prompt voice corresponding to the voice prompt information can be played through one or more sound units.

[0107] In the embodiment of the present application, the sound unit corresponding to the getting-off direction (called the target sound unit) can also be determined. For example, if the getting-off direction is the direction corresponding to the main driving position, the sound unit located at the main driving position can be determined as the target sound unit; in addition, the voice prompt information corresponding to the getting-off direction (called the target voice prompt information) can also be determined. For example, if the getting-off direction is the direction corresponding to the main driving position, the voice prompt information of the fourth obstacle whose fourth position information is located near the main driving position can be determined as the target voice prompt information; thereafter, the target voice prompt information can be played using the target sound unit; thereby helping the user to judge the environment around the vehicle more accurately.

[0108] For example, a sound unit disposed at the main driving position may be used to play voice prompt information corresponding to a fourth obstacle located near the main driving position.

[0109] In a specific implementation of an embodiment of the present application, in addition to playing voice prompt information, preset prompt sounds can also be played by each sound-emitting unit based on the obstacle type detection result; specifically, different prompt sounds can be determined for different obstacle type detection results so that the user can pay attention to the vehicle environment in a timely manner; for example, prompt sounds of different frequencies can be set for different fourth distances. For example, if the fourth distance of the fourth obstacle is small, that is, the fourth obstacle is close to the vehicle, it can be determined that the frequency of the corresponding prompt sound is high, so that the user's attention can be effectively attracted; if the fourth distance of the fourth obstacle is large, that is, the fourth obstacle is far from the vehicle, it can be determined that the frequency of the corresponding prompt sound is low; for another example, prompt sounds of different frequencies can be set for different obstacle types; for example, a prompt sound with a higher frequency can be set for dynamic objects such as pedestrians, bicycles, and electric vehicles, so as to effectively attract the user's attention; a prompt sound with a lower frequency can also be set for static objects such as roadblocks and curbs; thereby, the user can accurately judge the distance or type of the obstacle and other information based on the prompt sound, which can improve the user's perception of the vehicle environment and enhance the safety of getting off the vehicle.

[0110] In a specific implementation of an embodiment of the present application, the specific voice prompt method can be determined by the user; for example, the user can choose to play voice prompt information or prompt sounds on multiple speakers simultaneously; for another example, the user can choose to play different voice prompt information or prompt sounds on different speakers, for example, the user can choose to play the corresponding voice prompt information or prompt sound only at the corresponding get-off direction; for another example, the user can choose to play the same voice prompt information or prompt sound on each speaker to remind users in the entire vehicle.

[0111] It should be understood that in actual vehicle door opening warning applications, the above-mentioned information display process and / or voice prompt process can be selected according to actual needs. For example, only each display screen can be used to display the display information; for another example, only each sound unit can be used to play the prompt voice or prompt sound; for another example, each display screen can be used to display the display information at the same time, and each sound unit can be used to play the prompt voice or prompt sound.

[0112] In a specific implementation method of an embodiment of the present application, a side door opening warning detection result can also be obtained; specifically, the side door opening warning detection result is a detection result determined based on the image to be detected acquired by the side view camera acquisition device. The specific detection process can refer to any side door opening warning detection method in the prior art, and the present application is not limited to this; for example, the side view camera acquisition device can be used to acquire images of the side and rear of the vehicle to obtain a corresponding image to be detected; thereafter, the image to be detected can be preprocessed (for example, denoising, contrast enhancement, color correction) to obtain a processed image, and the processed image to be detected can be compared with a reference image in which a dangerous moving object exists to determine whether there is a dangerous moving object in the image to be detected; if the processed image to be detected has a high similarity with the reference image, it can be determined that there is a dangerous moving object in the image to be detected; if the processed image to be detected has a low similarity with the reference image, it can be determined that there is no dangerous moving object in the image to be detected; thereafter, the detection result of the dangerous moving object can be used as a side door opening warning detection result, and a door opening warning can be performed based on the side door opening warning detection result.

[0113] As an example, if the side door warning detection result indicates that there is a dangerous moving object behind the side of the vehicle, the corresponding display information can be determined based on the side door warning detection result; for example, the marking box and distance display text of the dangerous moving object can be determined, and the display information can be displayed using one or more display screens; if the side door warning detection result indicates that there is no dangerous moving object behind the side of the vehicle, only the image to be detected captured by the side view camera acquisition device can be displayed for the user to observe the environment behind the side of the vehicle.

[0114] As an example, if the side door warning detection result indicates that there is a dangerous moving object behind the side of the vehicle, one or more sound units can be used to play a preset prompt voice or prompt sound (for example, a prompt voice with the content "Danger behind the vehicle"); if the side door warning detection result indicates that there is no dangerous moving object behind the side of the vehicle, no voice prompt is required.

[0115] As an example, a vehicle door opening warning can also be performed based on the side door opening warning detection result and the obstacle type detection of the embodiment of the present application. Specifically, if the side door opening warning detection result indicates that there is a dangerous moving object behind the side of the vehicle, the vehicle door opening warning can be performed with reference to the above example; if the side door opening warning detection result indicates that there is no dangerous moving object behind the side of the vehicle, the vehicle door opening warning can be performed based on the obstacle type detection of the embodiment of the present application. The specific process can be referred to the above description and will not be repeated here.

[0116] In summary, in the embodiment of the present application, when the basic information of the vehicle meets the triggering condition of the door opening warning, the first distance measurement result and the second distance measurement result are determined respectively; the first distance measurement result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second distance measurement result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device; based on the first weight, the second weight, the first distance measurement result and the second distance measurement result, the third distance measurement result is determined; the first weight is the weight corresponding to the first distance measurement result, and the second weight is the weight corresponding to the second distance measurement result; based on the third distance measurement result, the obstacle type detection is performed on each image to be detected to obtain the obstacle type detection result; the door opening warning is performed based on the obstacle type detection result. Through the embodiment of the present application, the distance measurement result (first distance measurement result) determined by the ultrasonic radar and the distance measurement result (second distance measurement result) determined by the camera acquisition device can be combined, which can greatly improve the detection accuracy and detection distance in complex environments, accurately warn the vehicle door opening, and ensure the safety of the vehicle door opening.

[0117] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0118] Corresponding to the door opening warning method described in the above embodiment, Figure 7 A structural diagram of an embodiment of a door opening warning device provided in an embodiment of the present application is shown.

[0119] In an embodiment of the present application, a door opening warning device may be an electronic device, or a chip or circuit used in an electronic device, and the device includes:

[0120] The first determination module 701 is used to determine the first distance measurement result and the second distance measurement result respectively when the basic information of the vehicle meets the door opening warning triggering condition; the first distance measurement result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second distance measurement result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device;

[0121] A second determining module 702 is configured to determine a third ranging result based on a first weight, a second weight, the first ranging result, and the second ranging result; the first weight is a weight corresponding to the first ranging result, and the second weight is a weight corresponding to the second ranging result;

[0122] The detection module 703 is used to perform obstacle type detection on each image to be detected based on the third distance measurement result to obtain an obstacle type detection result; any of the images to be detected is an image captured by any of the camera acquisition devices;

[0123] The warning module 704 is used to issue a door opening warning based on the obstacle type detection result.

[0124] In a specific implementation of the embodiment of the present application, the first determining module includes:

[0125] A signal acquisition submodule, used to acquire each ultrasonic echo signal received by one or more ultrasonic radars;

[0126] A first information determination submodule, configured to determine the distance measurement information of each of the first obstacles based on each of the ultrasonic echo signals; the distance measurement information of any of the first obstacles includes the first position information and / or the first distance of any of the first obstacles;

[0127] A first result determination submodule, configured to determine the first ranging result based on the first position information and / or the first distance of each of the first obstacles;

[0128] An image acquisition submodule, used to acquire the images to be detected respectively acquired by the plurality of camera acquisition devices;

[0129] A second information determination submodule, configured to determine the distance measurement information of each second obstacle based on each adjacent frame of the to-be-detected image acquired by each of the camera acquisition devices; the distance measurement information of any second obstacle includes the second position information and / or the second distance of any second obstacle;

[0130] The second result determination submodule is used to determine the second ranging result based on the second position information and / or the second distance of each of the second obstacles.

[0131] In a specific implementation of the embodiment of the present application, the second determining module includes:

[0132] an obstacle determination submodule, configured to determine each third obstacle, each first specific obstacle, and each second specific obstacle; each of the third obstacles is an obstacle in the intersection of each of the first obstacles and each of the second obstacles, each of the first specific obstacles is an obstacle in the difference between each of the first obstacles and each of the second obstacles, and each of the second specific obstacles is an obstacle in the difference between each of the second obstacles and each of the first obstacles;

[0133] a weighted averaging submodule, configured to perform weighted averaging of the ranging information of the first obstacle and the ranging information of the second obstacle respectively corresponding to each of the third obstacles based on the first weight and the second weight, so as to obtain the ranging information of each of the third obstacles;

[0134] The third result determination submodule is used to determine the third ranging result based on the ranging information of each of the first unique obstacles, the ranging information of each of the second unique obstacles, and the ranging information of each of the third obstacles.

[0135] In a specific implementation of the embodiment of the present application, the detection module includes:

[0136] an area determination submodule, configured to determine, based on the ranging information of any fourth obstacle, an area to be detected corresponding to any fourth obstacle; any fourth obstacle is any obstacle among each of the first specific obstacles, each of the second specific obstacles, and each of the third obstacles;

[0137] A type detection submodule, used to perform obstacle type detection on the to-be-detected areas corresponding to each of the fourth obstacles, to obtain the obstacle type of each of the fourth obstacles;

[0138] The detection result determination submodule is used to determine the obstacle type detection result based on the fourth distance, the fourth position information and the obstacle type respectively corresponding to each of the fourth obstacles.

[0139] In a specific implementation of the embodiment of the present application, the device further includes:

[0140] A position determination module, used to determine one or more get-off positions based on the basic vehicle information;

[0141] Accordingly, the early warning module includes:

[0142] A display information determination submodule, used to determine various display information based on the obstacle type detection result;

[0143] An information display submodule, configured to display each of the display information using one or more display screens based on one or more of the alighting positions;

[0144] and / or,

[0145] A prompt information determination submodule, used to determine various voice prompt information based on the obstacle type detection result;

[0146] The voice playing submodule is used to play the prompt voice corresponding to each voice prompt information based on one or more getting-off directions using one or more sound-emitting units.

[0147] In a specific implementation of the embodiment of the present application, the information display submodule includes:

[0148] A display determination unit, used to determine the target display screen and target display information corresponding to each of the alighting positions;

[0149] An information display unit, used to use each of the target display screens to display the corresponding target display information respectively;

[0150] The prompt information determination submodule includes:

[0151] A prompt determination unit, used to determine the target sound units and target voice prompt information corresponding to each of the getting-off positions;

[0152] The voice playing unit is used to use each of the target sound producing units to play the corresponding target voice prompt information respectively.

[0153] In a specific implementation of the embodiment of the present application, the device further includes:

[0154] A result acquisition module, used to acquire a side door opening warning detection result; the side door opening warning detection result is determined based on multiple side view camera acquisition devices;

[0155] The warning module is also used to perform door opening warning based on the side door opening warning detection result.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0159] like Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned embodiments of the door opening warning method are implemented, for example Figure 3Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 7 The functions of modules 701 to 704 are shown.

[0160] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the electronic device 8.

[0161] Those skilled in the art will understand that Figure 8 It is only an example of the electronic device 8 and does not constitute a limitation of the electronic device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 8 may also include input and output devices, network access devices, buses, etc.

[0162] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0163] The memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. The memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 8. Further, the memory 81 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the electronic device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0164] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0165] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0166] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0167] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0171] The relevant user personal information that may be involved in each embodiment of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purpose of the business scenario, to process the personal information that the user actively provides during the use of the product / service or generated by the use of the product / service, as well as the personal information obtained with the user's authorization. The user personal information processed by the applicant will vary depending on the specific product / service scenario and must be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other relevant information. The applicant will treat the user's personal information and its processing with a high degree of diligence. The applicant attaches great importance to the security of user personal information, and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being unauthorized access, public disclosure, use, modification, damage or loss.

[0172] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A door opening warning method, characterized in that: include: When the basic information of the vehicle meets the door opening warning triggering condition, a first ranging result and a second ranging result are determined respectively; The first ranging result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second ranging result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device; Determine a third ranging result based on a first weight, a second weight, the first ranging result, and the second ranging result; the first weight is a weight corresponding to the first ranging result, and the second weight is a weight corresponding to the second ranging result; Performing obstacle type detection on each image to be detected based on the third distance measurement result to obtain an obstacle type detection result; any of the images to be detected is an image captured by any of the camera acquisition devices; A door opening warning is performed based on the obstacle type detection result.

2. The door opening warning method according to claim 1, characterized in that: The determining the first ranging result and the second ranging result respectively includes: Acquire each ultrasonic echo signal received by one or more ultrasonic radars; Based on each of the ultrasonic echo signals, determining the distance measurement information of each of the first obstacles; the distance measurement information of any of the first obstacles includes the first position information and / or the first distance of any of the first obstacles; Obtaining the first distance measurement result based on the first position information and / or the first distance of each of the first obstacles; Acquire the images to be detected respectively collected by the plurality of camera collection devices; Based on each of the adjacent frames of the to-be-detected images acquired by each of the camera acquisition devices, determining the ranging information of each of the second obstacles; the ranging information of any of the second obstacles includes the second position information and / or the second distance of any of the second obstacles; The second distance measurement result is determined based on the second position information and / or the second distance of each of the second obstacles.

3. The door opening warning method according to claim 2, characterized in that: The determining a third ranging result based on the first weight, the second weight, the first ranging result, and the second ranging result includes: Determine each third obstacle, each first specific obstacle, and each second specific obstacle; each third obstacle is an obstacle in the intersection of each first obstacle and each second obstacle, each first specific obstacle is an obstacle in the difference between each first obstacle and each second obstacle, and each second specific obstacle is an obstacle in the difference between each second obstacle and each first obstacle; Based on the first weight and the second weight, weighted averaging the ranging information of the first obstacle and the ranging information of the second obstacle respectively corresponding to each of the third obstacles to obtain the ranging information of each of the third obstacles; The third ranging result is determined based on the ranging information of each of the first unique obstacles, the ranging information of each of the second unique obstacles, and the ranging information of each of the third obstacles.

4. The door opening warning method according to claim 3, characterized in that: The performing obstacle type detection on each of the to-be-detected images based on the third ranging result to obtain an obstacle type detection result includes: Based on the ranging information of any fourth obstacle, determining the to-be-detected area corresponding to any fourth obstacle; any fourth obstacle is any obstacle among each of the first specific obstacles, each of the second specific obstacles and each of the third obstacles; Performing obstacle type detection on the to-be-detected areas respectively corresponding to the fourth obstacles to obtain the obstacle type of the fourth obstacles; The obstacle type detection result is determined based on the fourth distances, the fourth position information, and the obstacle types respectively corresponding to the fourth obstacles.

5. The door opening warning method according to claim 1, characterized in that: Before performing the door opening warning based on the obstacle type detection result, the method further includes: Determine one or more get-off locations based on the basic vehicle information; Accordingly, the door opening warning based on the obstacle type detection result includes: Determining various display information based on the obstacle type detection result; Based on one or more of the alighting positions, displaying each of the display information using one or more display screens; and / or, Determining various voice prompt information based on the obstacle type detection result; Based on one or more of the getting-off directions, one or more sound-emitting units are used to play the prompt voices corresponding to the respective voice prompt information.

6. The door opening warning method according to claim 5, characterized in that: The method of displaying each of the display information based on one or more of the getting-off positions using one or more display screens includes: Determine the target display screen and target display information corresponding to each of the alighting positions; Utilizing each of the target display screens to display the corresponding target display information respectively; The method of playing the prompt voice corresponding to each of the voice prompt information based on one or more of the getting-off directions by using one or more sound generating units includes: Determine the target sound units and target voice prompt information corresponding to each of the getting-off positions; Utilize each of the target sound emitting units to play the corresponding target voice prompt information respectively.

7. The door opening warning method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining a side door opening warning detection result; the side door opening warning detection result is determined based on multiple side view camera acquisition devices; A door opening warning is performed based on the side door opening warning detection result.

8. A door opening warning device, characterized in that: include: A first determination module is used to determine the first distance measurement result and the second distance measurement result respectively when the basic information of the vehicle meets the door opening warning triggering condition; The first ranging result is used to reflect the first position information and / or the first distance of each first obstacle measured by one or more ultrasonic radars deployed on the smart device, and the second ranging result is used to reflect the second position information and / or the second distance of each second obstacle measured by multiple camera acquisition devices deployed on the smart device; A second determining module, configured to determine a third ranging result based on the first weight, the second weight, the first ranging result, and the second ranging result; The first weight is a weight corresponding to the first ranging result, and the second weight is a weight corresponding to the second ranging result; A detection module, configured to perform obstacle type detection on each image to be detected based on the third distance measurement result to obtain an obstacle type detection result; any of the images to be detected is an image captured by any of the camera acquisition devices; The early warning module is used to issue a door opening early warning based on the obstacle type detection result.

9. A computer program product, characterized in that The invention comprises a computer program, and when the computer program is executed, the door opening warning method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the steps of the door opening warning method according to any one of claims 1 to 7.

Images