Vehicle window intelligent display method, system and apparatus, electronic device, and storage medium

Abstract

A vehicle window intelligent display method, comprising: collecting voice information, and parsing a pre-search target in voice and information of pre-search target features; parsing the pre-search target and the information of the pre-search target features into information of image elements; an on-board camera collecting a vehicle surrounding image; according to the visual angle of a driver, correspondingly displaying the collected vehicle surrounding image on a vehicle window; segmenting the collected vehicle surrounding image to generate a plurality of object images waiting for recognition; according to the pre-search target and the image elements of the pre-search target features, scanning the vehicle surrounding image collected in real time; and if the object image waiting for recognition which conforms to the image elements of the pre-search target features is scanned, adding a label to the object image waiting for recognition which is correspondingly displayed on the vehicle window, so as to prompt that same is a pre-search target. According to the method, a search target can be displayed on a vehicle window, so that the burden of searching a target is reduced, and driving safety is maintained. Further provided are a vehicle window intelligent display system, a vehicle window intelligent display apparatus, an electronic device, and a storage medium.

Description

Vehicle window intelligent display method, system, device, electronic device and storage medium Technical Field

[0001] The present application relates to the field of display, and in particular to a vehicle window intelligent display method, a vehicle window intelligent display system, a vehicle window intelligent display device, an electronic device and a storage medium. Background Art

[0002] When driving, users often encounter scenarios where they need to search for people, street shops, office buildings, road signs and other targets outside the car while driving, resulting in: the line of sight often deviating from the road ahead for a long time, ignoring the road conditions ahead, and easily causing traffic accidents; when using the eyes to identify and search for targets outside the car, the brain will allocate a lot of energy to identifying and locking targets, neglecting driving operations, and the ability to respond to emergencies is reduced, seriously affecting driving safety; because the line of sight needs to continuously search for targets outside the car, the speed of the car will inevitably be reduced for a long time, which can easily cause congestion of vehicles behind.

[0003] Therefore, there is a need for a solution that can automatically assist the driver in searching for objects around the vehicle based on voice commands, so that the driver can keep his attention on the road conditions ahead.

[0004] Summary of the Invention

[0005] The purpose of the present invention is to provide a vehicle window intelligent display method, a vehicle window intelligent display system, a vehicle window intelligent display device, an electronic device and a storage medium to solve at least one of the above-mentioned technical problems.

[0006] The present invention provides the following solutions:

[0007] According to one aspect of the present invention, a vehicle window intelligent display method is provided, the vehicle window intelligent display method comprising:

[0008] Collect voice information and analyze the information of pre-search targets and pre-search target features in the voice;

[0009] Parsing the pre-search target and pre-search target feature information into information of image elements;

[0010] The on-board camera collects images around the vehicle;

[0011] The collected images around the vehicle are displayed on the window according to the driver's perspective;

[0012] Segmenting the collected images around the vehicle to generate multiple images of objects to be identified;

[0013] Scanning the real-time collected image of the vehicle surroundings according to the pre-search target and the image elements of the pre-search target features;

[0014] If an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned, a label is added to the image of the object to be identified displayed on the window, indicating that it is the pre-search target.

[0015] Furthermore, the collecting of voice information and parsing of information about the pre-search target and pre-search target features in the voice include:

[0016] Parse sentences containing feature information and target information in speech;

[0017] The parsed target is pointed to according to the parsed features, and the pre-search target and the features of the pre-search target are determined.

[0018] Furthermore, the step of parsing the pre-search target and the pre-search target feature information into information of image elements includes:

[0019] According to the pre-searched target described by the voice, a sample image corresponding to the pre-searched target is selected from a preset database;

[0020] Based on the features of the pre-searched target described by the speech, pre-compared image features are marked on the example image;

[0021] Determine the degree of similarity between the image of the object to be identified and the example image based on the example image and the marked pre-compared image features;

[0022] If the image of the object to be identified and the object in the example image are classified as the same type of object, then extracting image features of the image of the object to be identified based on the pre-compared image features;

[0023] If the similarity between the image features of the object to be identified and the image features of the example image exceeds a preset threshold, the object to be identified is marked and displayed as a pre-search target.

[0024] Furthermore, displaying the collected vehicle surrounding image on the vehicle window according to the driver's perspective includes:

[0025] Based on the segmented and collected images around the vehicle, multiple images of the object to be identified are generated and the images are processed in layers;

[0026] The layered processing includes displaying the outline of the image of the object to be identified on the car window;

[0027] Add annotations based on the image of the object to be identified displayed on the vehicle window, and display the outline of the image of the object to be identified in a preset display style.

[0028] Furthermore, judging the similarity between the to-be-identified object image and the example image based on the example image and the marked pre-compared image features includes:

[0029] Assign values ​​to image features according to the degree to which they can be identified;

[0030] According to the assignment of the pre-search target features, the similarity between the image of the object to be identified and the example image is judged.

[0031] Furthermore, if the image of the object to be identified and the object in the example image are classified as the same type of object, extracting image features of the image of the object to be identified based on the pre-compared image features includes:

[0032] Classify the types of objects in the image of the object to be identified and the sample image according to the preset first-level image features;

[0033] Determine the similarity between the image of the object to be identified and the sample image based on the preset second-level image features;

[0034] Wherein, the second-level image features are preset according to the pre-compared image features.

[0035] According to two aspects of the present invention, there is provided a vehicle window intelligent display system, the vehicle window intelligent display system comprising: the target recognition module and the target display module;

[0036] The target recognition module is used to identify similar objects to be recognized in the images around the vehicle based on the pre-searched targets analyzed in the speech;

[0037] The target display module is used to display the vehicle surrounding image on the car window corresponding to the driver's perspective, including adding a label similar to the pre-searched target on the car window display corresponding to the image of the object to be identified according to the pre-searched target analyzed in the voice.

[0038] According to three aspects of the present invention, a vehicle window intelligent display device is provided, the vehicle window intelligent display device comprising:

[0039] The voice acquisition module is used to collect voice information and analyze the information of pre-search targets and pre-search target features in the voice;

[0040] An image conversion module, configured to parse the pre-search target and pre-search target feature information into information of image elements;

[0041] Image acquisition module, used for the vehicle camera to collect images around the vehicle;

[0042] An image display module is used to display the collected images of the vehicle's surroundings on the vehicle window according to the driver's perspective;

[0043] An image segmentation module is used to segment the collected images around the vehicle and generate multiple images of objects to be identified;

[0044] An image scanning module, configured to scan the real-time collected image of the vehicle's surroundings based on the pre-search target and the image elements of the pre-search target's features;

[0045] The image annotation module is used to add an annotation to the image of the object to be identified displayed on the vehicle window if an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned, indicating that it is the pre-search target.

[0046] According to four aspects of the present invention, there is provided an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0047] A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the vehicle window intelligent display method.

[0048] According to five aspects of the present invention, a computer-readable storage medium is provided, comprising: a computer program that can be executed by an electronic device is stored therein, and when the computer program runs on the electronic device, the electronic device executes the steps of the vehicle window intelligent display method.

[0049] Through the above solution, the following beneficial technical effects are achieved:

[0050] This application collects the driver's voice, analyzes the target the driver wants to find, and displays the target the driver may want to find on the car window, so that the driver can keep paying attention to the road conditions while reducing the burden of finding the target, allowing the driver to improve the efficiency of finding the target while maintaining vehicle driving safety.

[0051] This application displays objects around the vehicle on the window according to the driver's perspective, so that the objects outside the vehicle currently observed by the driver are in a "virtual" correspondence with the real-time collected images, making it easier for the driver to control the driving trajectory and pay attention to objects outside the vehicle, showing a position change relationship corresponding to the body sensation, and reducing the driver's burden of interpreting image data.

[0052] This application displays the outline of the image on the car window, retains most of the transparent part of the car window, so that the driver's vision of driving the vehicle is kept in the first perspective, and the outline corresponds to the real observed objects, so that the driver is affected by the image displayed on the car window, and maintains absolute control over the car computer's driving vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] FIG1 is a flow chart of a vehicle window intelligent display method provided by one or more embodiments of the present invention.

[0054] FIG2 is a structural diagram of a vehicle window intelligent display device provided by one or more embodiments of the present invention.

[0055] FIG3 is a structural diagram of a vehicle window intelligent display system provided by one or more embodiments of the present invention.

[0056] FIG4 is a schematic diagram of the main process of displaying a target on a vehicle window according to voice according to a specific embodiment of the present invention.

[0057] FIG5 is a schematic diagram of a system for displaying objects on a vehicle window according to speech according to a specific embodiment of the present invention.

[0058] FIG6 is a schematic diagram of a target detection process according to a specific embodiment of the present invention.

[0059] FIG7 is a schematic diagram of a target detection process using a multimodal model according to a specific embodiment of the present invention.

[0060] FIG8 is a schematic diagram of a main process change using a multimodal model according to a specific embodiment of the present invention.

[0061] FIG9 is a block diagram of an electronic device structure of a vehicle window intelligent display method provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] FIG1 is a flow chart of a vehicle window intelligent display method provided by one or more embodiments of the present invention.

[0064] As shown in FIG1 , the vehicle window intelligent display method includes:

[0065] Step S1, collecting voice information, parsing the information of pre-search target and pre-search target features in the voice;

[0066] Step S2, parsing the pre-search target and the pre-search target feature information into information of image elements;

[0067] Step S3, the vehicle-mounted camera collects images around the vehicle;

[0068] Step S4, displaying the collected image of the vehicle's surroundings on the vehicle window according to the driver's perspective;

[0069] Step S5, segmenting the collected image around the vehicle to generate multiple images of objects to be identified;

[0070] Step S6, scanning the real-time collected image of the vehicle surroundings according to the pre-search target and the image elements of the pre-search target features;

[0071] In step S7, if an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned, a label is added to the image of the object to be identified displayed on the vehicle window, indicating that it is the pre-search target.

[0072] Through the above solution, the following beneficial technical effects are achieved:

[0073] This application collects the driver's voice, analyzes the target the driver wants to find, and displays the target the driver may want to find on the car window, so that the driver can keep paying attention to the road conditions while reducing the burden of finding the target, allowing the driver to improve the efficiency of finding the target while maintaining vehicle driving safety.

[0074] This application displays objects around the vehicle on the window according to the driver's perspective, so that the objects outside the vehicle currently observed by the driver are in a "virtual" correspondence with the real-time collected images, making it easier for the driver to control the driving trajectory and pay attention to objects outside the vehicle, showing a position change relationship corresponding to the body sensation, and reducing the driver's burden of interpreting image data.

[0075] This application displays the outline of the image on the car window, retains most of the transparent part of the car window, so that the driver's vision of driving the vehicle is kept in the first perspective, and the outline corresponds to the real observed objects, so that the driver is affected by the image displayed on the car window, and maintains absolute control over the car computer's driving vision.

[0076] Specifically, while driving, drivers may need to identify objects outside the vehicle, such as looking for a guest or a restaurant. People may stand on the roadside to avoid vehicles, and restaurants may be located on both sides of the road, facing a different direction than the driver's forward gaze. Diverting attention away from the vehicle's forward direction is dangerous.

[0077] Car computers are often equipped with a voice recognition module to identify voice commands such as playing songs and navigating to different locations. Similarly, drivers can use the car computer's voice recognition function to analyze speech and identify the target they are looking for. However, unlike traditional voice recognition, the target the driver is looking for is an object with a "shape." Therefore, the "target" described in the voice must be converted into information in the form of image data, and image comparison is performed to assist the driver in finding the target. During the voice analysis process, in addition to specifying the target to be found, sufficient features must be added to the target so that the image analysis has sufficient image features for comparison.

[0078] The vehicle-mounted camera can be used to record objects outside the vehicle, generate a video file, and compare the target that the driver is looking for in the video frame.

[0079] Real-time video footage recorded by onboard cameras needs to be displayed on the car window to help keep the driver's eyes focused on the road ahead. However, the entire video cannot be displayed on the car window, as this would obstruct the driver's view and compromise driving safety.

[0080] The outline of an object can be displayed on the car window, while keeping most of the windows from displaying image information, ensuring that the driver can see outside the car through the window.

[0081] For example, existing window display systems, such as HUD (Head-up-Display) systems, can only project the outlines of objects in the image onto the window.

[0082] For example, by identifying nouns and adjectives in a speech sentence, the noun can be used as a pre-search target, and the adjective describing the noun can be used as the pre-search target feature. If the pre-search target and the pre-search target feature match, the pre-search target can be converted into image data. For example, an image corresponding to the pre-search target can be found in an image database and used as a sample. Using the pre-search target features, the sample image can then be annotated with pre-comparison image features.

[0083] The real-time image of the vehicle's surroundings can be segmented into independent image units, each representing an image of an object to be identified. The image of the object to be identified is used as the target for identification, and it is determined whether the pre-search target that the driver wants to find exists in the image of the object to be identified.

[0084] The image of the object to be identified can be annotated according to the pre-compared image features, so that the pre-search target image in the image of the object to be identified, that is, the pre-search target in the voice command, can be filtered according to the image features.

[0085] When displayed on the car window, the image of the object to be identified that is similar to the pre-searched target image is marked, for example, different colors are displayed on the outline.

[0086] In this embodiment, collecting voice information and analyzing the information of the pre-search target and the pre-search target features in the voice include:

[0087] Parse sentences containing feature information and target information in speech;

[0088] The parsed target is pointed to according to the parsed features, and the pre-search target and the features of the pre-search target are determined.

[0089] Specifically, for example, the user's voice input, or unstructured request text, is converted into a structured semantic representation. This structured semantic representation consists of three parts: domain, intent, and slot. The domain is the scope of the user's request, the intent is the type of the user's request, and the slot is the entity describing the user's request. For example, if a user requests "Help me find a tall female student with a ponytail and a red top," the domain is "find target" (i.e., "help me" is the triggering instruction), the intent is "person" (i.e., the pre-search target), and the slot is "hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student" (i.e., the pre-search target features).

[0090] In this embodiment, the information of parsing the pre-search target and the pre-search target feature information into image elements includes:

[0091] According to the pre-searched target described by the voice, a sample image corresponding to the pre-searched target is selected from a preset database;

[0092] Based on the features of the pre-searched target described by the speech, pre-compared image features are marked on the example image;

[0093] Determine the degree of similarity between the image of the object to be identified and the example image based on the example image and the marked pre-compared image features;

[0094] If the image of the object to be identified and the object in the example image are classified as the same type of object, then extracting image features of the image of the object to be identified based on the pre-compared image features;

[0095] If the similarity between the image features of the object to be identified and the image features of the example image exceeds a preset threshold, the object to be identified is marked and displayed as a pre-search target.

[0096] Specifically, while speech parsing into text is readable, it cannot be directly used for image feature comparison. The pre-search target can be mapped to a sample image in the image database. For example, if the query asks "Find me a tall female student with a ponytail and a red top," and the target is a person, an image of a person in the image database can be used as a sample. Furthermore, if the target has the features "hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student," the image to be identified can be filtered based on the pre-compared image features.

[0097] To reduce the computational complexity of recognition, feature comparison is performed in a hierarchical manner. First, objects are classified according to general object types, and then the images to be identified are compared against pre-compared image features. While the defined recognition features for object classification vary between systems, their impact on the results is minimal, and the images to be identified do not necessarily need to be compared against pre-compared image features. For example, for image features representing "people," overly stringent feature comparison is not required. However, if features such as "hairstyle = ponytail, top = red, height = tall, gender = female, occupation = student" are specifically specified in the user's voice command, they need to be compared against the images to be identified.

[0098] It's possible that the driver's voice command may contain a mistranslation of one or two pre-search target features, resulting in a discrepancy between the features and the actual image of the object being identified. Alternatively, the user may be unsure of their memory and may be ambiguous. Pre-search target features can be pre-assigned values, and the window display can be marked based on whether the combined score of multiple features exceeds a preset threshold.

[0099] In this embodiment, the collected image of the vehicle's surroundings corresponding to the driver's perspective and displayed on the vehicle window includes:

[0100] Based on the segmented and collected images around the vehicle, multiple images of the object to be identified are generated and the images are processed in layers;

[0101] The layered processing includes displaying the outline of the image of the object to be identified on the car window;

[0102] Add annotations based on the image of the object to be identified displayed on the vehicle window, and display the outline of the image of the object to be identified in a preset display style.

[0103] Specifically, to find the pre-search target in the image around the vehicle, it is necessary to first segment the image around the vehicle. Since the on-board camera records images frame by frame, each image may contain multiple images of the object to be identified. If the identification features are directly marked on them, it will be impossible to distinguish which part the image features belong to during the subsequent comparison process. Therefore, it is necessary to segment the image around the vehicle according to the individual images of the object to be identified, so that the pre-comparison image features can be easily marked on them. Maintaining a one-to-one image feature comparison between the pre-search target and the image of the object to be identified can avoid interference from other image features, making the recognition process more accurate.

[0104] The collected images of the vehicle's surroundings correspond to the driver's perspective and are displayed on the car window. The image of the object to be identified is marked with contour lines. The actual effect is to mark the contour lines on the objects outside the car that the driver actually observes.

[0105] The driver can identify the person outside the car based on the outline as the person they're looking for. Displaying the outline on a standard screen requires the driver to switch from their current perspective to the camera's. Switching between the two perspectives requires a strong ability to switch spatial perspectives, increasing the driver's mental workload. Therefore, displaying the outline on the car window provides a more user-friendly experience for the driver.

[0106] In this embodiment, judging the similarity between the to-be-identified object image and the example image based on the example image and the marked pre-compared image features includes:

[0107] Assign values ​​to image features according to the degree to which they can be identified;

[0108] According to the assignment of the pre-search target features, the similarity between the image of the object to be identified and the example image is judged.

[0109] Specifically, it's possible that the pre-search target features described in the driver's voice don't fully match the features of the actual object image to be identified. For example, a driver might want to pick someone up at the bus stop, but they don't know the person's current attire. Therefore, the pre-search target features provided in the voice description are subject to significant uncertainty. For example, it's difficult to confirm whether the features "hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student" have changed, resulting in a low degree of reliability. Therefore, a higher weight could be assigned to the features "height = tall, gender = female, occupation = student" and a lower weight to "hairstyle = ponytail, top = red top." Through comprehensive scoring, we can derive data on the degree of similarity.

[0110] You can also pre-search target features based on their ease of visualization and assign values ​​to them. For example, the feature "occupation = student" is difficult to represent using image data, but the features "hairstyle = ponytail, top = red top, height = tall, gender = female" are easier to represent using image data and have a slightly higher reliability. Therefore, you can assign a lower weight to the feature "occupation = student" and a higher weight to the features "hairstyle = ponytail, top = red top, height = tall, gender = female."

[0111] In this embodiment, if the image of the object to be identified and the object in the example image are classified as the same type of object, extracting the image features of the image of the object to be identified based on the pre-compared image features includes:

[0112] Classify the types of objects in the image of the object to be identified and the sample image according to the preset first-level image features;

[0113] Determine the similarity between the image of the object to be identified and the sample image based on the preset second-level image features;

[0114] Wherein, the second-level image features are preset according to the pre-compared image features.

[0115] Specifically, the preset first-level image features provide a rough classification between the target image and the example image, preventing overly detailed features from overshadowing the underlying features. For example, a "person" might be a male in the example image, while the target image might be a female. This does not prevent both from being considered as "person" types for subsequent detailed comparison. However, if the fundamental "person" feature is overshadowed by detailed features such as clothing, hairstyle, and color, the comparison results may be biased and the key points may be lost.

[0116] You can first find relatively easy and basic types such as "people" or "women" to perform first-level image feature recognition, narrow the scope of comparison features, and then perform second-level image feature comparison.

[0117] Second-level image features can directly utilize pre-compared image features. This reduces computational effort while increasing recognition accuracy, while narrowing the comparison range using first-level image features. For example, if "object = human, hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student" is directly used for second-level image features, a recognition error for "object = human" could lead to errors in the overall recognition result. For example, if "object = human" is used for first-level image features, and "hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student" is used for second-level image features, even if individual second-level image features are missing, this will not affect the recognition of the "human" feature. However, if "object = human" is used for first-level image features, it has a very low error rate.

[0118] Of course, people aren't the only first-level image features. The scope of first-level image features can be temporarily adjusted. For example, on sunny days, clothing color recognition accuracy is high and reliability is high, so clothing color can be used as the first-level image feature. However, features with excessive details, such as clothing patterns, are not suitable as first-level image features.

[0119] FIG2 is a structural diagram of a vehicle window intelligent display device provided by one or more embodiments of the present invention.

[0120] As shown in Figure 2, the car window intelligent display device includes: a voice acquisition module, an image conversion module, an image acquisition module, an image display module, an image segmentation module, an image scanning module, and an image annotation module;

[0121] The voice acquisition module is used to collect voice information and analyze the information of pre-search targets and pre-search target features in the voice;

[0122] An image conversion module is used to parse the pre-search target and pre-search target feature information into information of image elements;

[0123] Image acquisition module, used for the vehicle camera to collect images around the vehicle;

[0124] An image display module is used to display the collected images of the vehicle's surroundings on the vehicle window according to the driver's perspective;

[0125] An image segmentation module is used to segment the collected images around the vehicle and generate multiple images of objects to be identified;

[0126] An image scanning module is used to scan the real-time collected image around the vehicle based on the pre-search target and the image elements of the pre-search target features;

[0127] The image annotation module is used to add an annotation to the image of the object to be identified displayed on the vehicle window if an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned, indicating that it is the pre-search target.

[0128] It is worth noting that although the present system only discloses a voice acquisition module, an image conversion module, an image acquisition module, an image display module, an image segmentation module, an image scanning module, and an image annotation module, it does not mean that the present device is limited to the above-mentioned basic functional modules. Rather, what the present invention wants to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the existing technology to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the scope of protection of the claims of the present invention is limited to the above-mentioned basic functional modules.

[0129] FIG3 is a structural diagram of a vehicle window intelligent display system provided by one or more embodiments of the present invention.

[0130] As shown in Figure 3, the car window intelligent display system includes: a target recognition module and a target display module;

[0131] The target recognition module is used to identify similar objects to be recognized in the images around the vehicle based on the pre-searched targets analyzed in the speech;

[0132] The target display module is used to display the vehicle surrounding image on the car window corresponding to the driver's perspective, including adding a label similar to the pre-searched target on the car window display corresponding to the image of the object to be identified according to the pre-searched target analyzed in the voice.

[0133] Specifically, the pre-search target and pre-search target features of speech recognition are text data and need to be converted into image data before they can be used for comparative recognition of the collected images around the vehicle.

[0134] In terms of display method, only by displaying on the car window can an effect that cannot be achieved by other screen displays be achieved, that is, the objects outside the car currently observed by the driver are in a "virtual" corresponding relationship with the real-time collected images, which makes it convenient for the driver to control the driving trajectory and pay attention to objects outside the car, showing a position change relationship corresponding to the body sensation, reducing the driver's burden of interpreting image data.

[0135] Of course, vehicle windows aren't limited to the front window; depending on the camera's field of view, the image of the object to be identified can also be displayed on the side windows. While displaying the image of the object to be identified on the side windows hinders the driver's forward observation of road conditions, the automated image recognition process doesn't require the driver to constantly look at the side windows. Confirmation is only required when the image of the object to be identified matches the pre-searched target image.

[0136] Of course, displaying the image of the object to be identified on the vehicle's side window requires a corresponding camera angle. For example, when passing through an alley entrance, the image of the object to be identified in the image around the vehicle is captured as a pre-search target. At this moment, the image of the object to be identified is annotated and displayed on the vehicle's side window at the appropriate time.

[0137] A sound warning can be issued based on the scanned image of the object to be identified that meets the pre-searched target feature image elements, further reducing the visual burden and preventing the missed reminder information displayed on the window with the marked contour line due to the dynamic driving of the vehicle.

[0138] FIG4 is a schematic diagram of the main process of displaying a target on a vehicle window according to voice according to a specific embodiment of the present invention.

[0139] FIG5 is a schematic diagram of a system for displaying objects on a vehicle window according to speech according to a specific embodiment of the present invention.

[0140] FIG6 is a schematic diagram of a target detection process according to a specific embodiment of the present invention.

[0141] FIG7 is a schematic diagram of a target detection process using a multimodal model according to a specific embodiment of the present invention.

[0142] FIG8 is a schematic diagram of a main process change using a multimodal model according to a specific embodiment of the present invention.

[0143] As shown in Figures 4 and 5, the system follows a process. The driver speaks a voice command to search for a target, such as "Help me find a tall female student with a ponytail and a red top" or "Help me find a delicious Sichuan restaurant." The voice acquisition unit inputs these voice commands into the speech recognition module (which includes a fully trained speech recognition model and intent understanding model). The speech recognition module determines whether the driver intends to search for the target outside the vehicle. If so, it outputs keywords describing the user's request entity. The keywords describing the driver's request entity are input into the target matching module (including matching algorithm). At the same time, the off-vehicle image acquisition unit (including camera, radar, etc.) starts to collect video images within a certain range in front of the vehicle in real time and inputs them into the target detection module. The target detection module (including a well-trained semantic segmentation model, image classification model, and feature-to-text model) extracts video frames from the collected video images for semantic segmentation. The image classification algorithm performs category labeling on all targets extracted by semantic segmentation, selects targets of the same category as the driver's search intention (can be multiple), and further classifies the targets (the classification method is consistent with the classification of the slots of the voice intent model). The target image classification results (i.e., image features) are passed through the feature-to-text model to output all feature keywords of the target, and then the feature keywords are also input into the target matching module. The target matching module compares the target's characteristic keywords with the keywords describing the driver's requested entity and calculates a weighted score. When the score reaches the set threshold, the target is considered to meet the driver's search intention; at the same time, all target categories output by the target detection module are input into the 3D model library, and the 3D models of each target category are retrieved; the display module (including 3D rendering software) uses the 3D models screened by the model library and the three-dimensional position information of each target relative to the vehicle obtained by the external image acquisition unit for real-time 3D environment reconstruction of the image captured by the image acquisition unit, and displays it on the driver's front window (it can be a HUD solution, or the window glass itself can be used as a screen for display), and uses the relative position information and category information of the target that meets the search intention to determine the position of the target in the 3D model, and marks the 3D model of the target that meets the search intention with a specific color. The driver determines the direction and distance of the target by the relative position relationship between the color-marked target model on the window and the vehicle model and other surrounding reference object models. Then the driver moves his eyes to the location of the target in the real environment and makes a final manual judgment to determine whether the search target has appeared. If it is not the target to be found, the driver continues to wait for the next target marking, or repeats the voice command containing more target details to initiate a new round of search.

[0144] As shown in Figure 6, the target detection module extracts video frames from the captured video images for semantic segmentation; the image classification model labels all targets extracted by semantic segmentation; selects targets of the same category as the driver's search intention (can be multiple) and further classifies the targets; the target image classification results (image features) are passed through the feature-to-text model to output all feature keywords of the target.

[0145] In the speech recognition model, intent understanding converts the unstructured user request text into a structured semantic representation. This structured semantic representation consists of three parts: domain, intent, and slot. The domain is the scope of the user request, the intent is the type of request, and the slot is the entity describing the request. For example, a user requests, "Help me find a tall, female student with a ponytail and a red top." The domain is "find a target," the intent is "person," and the slots are "hairstyle = ponytail, top = red top, height = tall, gender = female, occupation = student." In this system, the "domain" only has two categories: "find a target" and "other." Therefore, intent understanding primarily focuses on intent classification and entity recognition. Common approaches to intent understanding models include, but are not limited to, rule-based and deep learning-based methods. Based on common scenarios involving finding objects outside a vehicle while driving, intents can be categorized as people, cars, animals, shops, restaurants, attractions, traffic signs, brand logos, and more. Different slots are defined for different intents. For example, the slots for the "person" intent include hairstyle, top, bottom, shoes, hat, height, body type, gender, age, occupation, glasses, accessories, posture, and items carried. The intent understanding model ultimately extracts slots and outputs keywords that describe the entity being requested.

[0146] In the object matching module, a matching algorithm is used for quantification. The speech recognition module outputs keywords describing the entity requested by the user, such as: Hairstyle = Ponytail, Top = Red Top, Height = Tall, Gender = Female, Occupation = Student. The object detection module outputs the characteristic keywords of all objects in the image. Suppose there are two objects in the image classified as "people." The characteristic keywords of the first object are: Hairstyle = Ponytail, Top = Red Top, Height = Tall, Gender = Female, Occupation = Unknown. The characteristic keywords of the second object are: Hairstyle = Ponytail, Top = Pink Top, Height = Tall, Gender = Female, Occupation = Student. For the "person" category, feature weights are defined as: "gender" (10%), "hairstyle" (10%), "tops" (10%), "height" (5%), "occupation" (5%), "bottoms" (10%), "shoes" (5%), "hat" (10%), "body type" (5%), "age" (10%), "glasses" (5%), "posture" (5%), "accessories" (5%), and "carried items" (5%). The keywords describing the user's requested entity output by the speech recognition module are compared with the feature keywords of each object in the image output by the object detection module. A match is scored as 1, and a mismatch is scored as 0. The matching score for each keyword is then multiplied by its weight and summed to calculate the final score. Because the keywords describing the user's requested entity may not cover all slots corresponding to the intent, the score must be divided by the ratio of the weights of the covered keywords to the total weight (100%). For the first target, the comparison results are as follows: a hairstyle match is scored as 1 point multiplied by the hairstyle's weight of 10%, a top match is scored as 1 point multiplied by the top's weight of 10%, a height match is scored as 1 point multiplied by the height's weight of 5%, a gender match is scored as 1 point multiplied by the gender's weight of 10%, and an occupation mismatch is scored as 0 point. Adding these values ​​together gives a total of 0.35, which is divided by the weight of the covered keywords, which is 0.4, for a final score of 0.875. For the second target, the comparison results are as follows: a hairstyle match is scored as 1 point multiplied by the hairstyle's weight of 10%, a top mismatch is scored as 0 point, a height match is scored as 1 point multiplied by the height's weight of 5%, a gender match is scored as 1 point multiplied by the gender's weight of 10%, and an occupation match is scored as 1 point multiplied by the occupation's weight of 5%. Adding these values ​​together gives a total of 0.30, which is divided by the weight of the covered keywords, which is 0.4, for a final score of 0.75. If the threshold is set to 0.8, and the score of the first target is 0.875>0.8, it is determined to match the search intent; if the score of the second target is 0.75<0.8, it is determined to not match the search intent.

[0147] As shown in Figure 7, the object matching module can utilize more than just matching algorithms; it can also employ a multimodal model. If a multimodal model is used, the object detection module workflow does not require the output of feature keywords. The object detection module extracts frames from the captured video imagery for semantic segmentation; the image classification module labels all objects identified by semantic segmentation.

[0148] The keywords describing the user's requested entity output by the speech recognition module and all target images output by the target detection module are input into the target matching module (including the multimodal model). Similarity is calculated and a matching score is obtained. Targets whose scores reach the set threshold are considered to meet the search intent. The multimodal model performs a semantic similarity rating between images and text, and the scoring criteria are based on semantic text similarity. If a multimodal model is used, steps 4 and 5 in Figure 6 are adjusted, as shown in Figure 8.

[0149] In another embodiment, an actual application scenario is described. For example, Xiao Zhang is driving to pick up a client from out of town and agrees to meet on a certain road. Before departure, he asks the client what he is wearing today. The client says he is wearing a dark blue suit, a red tie, and carrying a brown leather bag. Because the agreed-upon road section is in a relatively busy area, as the vehicle is about to enter the target road section, Xiao Zhang issues a voice command to the vehicle, "Help me find a man wearing a dark blue suit, a red tie, and carrying a brown leather bag." The intent understanding model converts the above command into a structured semantic representation, with the domain being "find target," the intent being "person," and the slots being "shirt = dark blue suit, gender = male, age = adult, items carried = brown leather bag, accessories = red tie." The vehicle immediately entered the intelligent target recognition mode. When the vehicle found that the target that met the search intention was on the right front roadside, the 3D model diagram on the driver's front window showed that the target was 20 meters away from the right front roadside of the road on which the vehicle was traveling. Xiao Zhang turned his sight to the right side of the front window according to the position relationship of each reference object in the 3D diagram, and quickly found the customer he was looking for. Xiao Zhang parked the car safely and smoothly next to the customer.

[0150] In another embodiment, a practical application scenario is described. For example, Xiao Zhang is going to dinner with a blind date he's just met. The restaurant she's chosen is called "Haoweidao Sichuan Restaurant" on Beiyuan Road. Xiao Zhang has never been to this restaurant. After setting off, Xiao Zhang sends the command to the vehicle, "Help me find Haoweidao Sichuan Restaurant." The intent understanding model converts this command into a structured semantic representation, with the domain "find target," the intent "restaurant," and the slot "name = Haoweidao Sichuan Restaurant." Shortly after starting the journey, the 3D environment model on the car window marks the target location. Sure enough, it's Haoweidao Sichuan Restaurant, but it's quite far from Beiyuan Road. Xiao Zhang wonders, "Whoa, this restaurant is also a chain? There's one so close to my home, and I've never been there before!" When the vehicle reached Beiyuan Road, the target location was marked on the car window again. This should be the restaurant that the girl booked. Although the navigation prompts that it has arrived near the destination and automatically ends the navigation, the restaurant is actually located relatively close to the main road and is not easy to find. Fortunately, the intelligent search system helps him keep an eye on the outside of the car at all times. Otherwise, if he passed by it and then turned around at a farther place, he might have missed the agreed time.

[0151] FIG9 is a block diagram of an electronic device structure of a vehicle window intelligent display method provided by one or more embodiments of the present invention.

[0152] As shown in FIG9 , the present application provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0153] A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of a vehicle window intelligent display method.

[0154] The present application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a vehicle window intelligent display method.

[0155] The present application also provides a vehicle, comprising:

[0156] An electronic device for implementing the steps of the vehicle window intelligent display method;

[0157] A processor, wherein the processor runs a program and, when the program runs, executes the steps of the vehicle window intelligent display method based on data output by the electronic device;

[0158] The storage medium is used to store a program, and when the program is running, it executes the steps of the vehicle window intelligent display method for data output from the electronic device.

[0159] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0160] The electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory. The operating system can be any one or more computer operating systems that control electronic devices through processes, such as the Linux operating system, the Unix operating system, the Android operating system, the iOS operating system, or the Windows operating system. In the embodiments of the present invention, the electronic device can be a handheld device such as a smartphone or a tablet computer, or an electronic device such as a desktop computer or a portable computer, which is not particularly limited in the embodiments of the present invention.

[0161] The execution subject of the electronic device control in the embodiment of the present invention can be an electronic device, or a functional module in the electronic device that can call a program and execute the program. The electronic device can obtain the firmware corresponding to the storage medium. The firmware corresponding to the storage medium is provided by the supplier. The firmware corresponding to different storage media can be the same or different, and is not limited here. After the electronic device obtains the firmware corresponding to the storage medium, it can write the firmware corresponding to the storage medium into the storage medium, specifically, burn the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technology and will not be described in detail in the embodiment of the present invention.

[0162] The electronic device can also obtain a reset command corresponding to the storage medium. The reset command corresponding to the storage medium is provided by the supplier. The reset commands corresponding to different storage media can be the same or different, and are not limited here.

[0163] In this case, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written. The electronic device can respond to the reset command corresponding to the storage medium in which the corresponding firmware is written, thereby resetting the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented in the existing technology and will not be described in detail in the embodiments of the present invention.

[0164] For the convenience of description, the above devices are described as various units and modules according to their functions. Of course, when implementing this application, the functions of each unit and module can be implemented in the same or multiple software and / or hardware.

[0165] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with those in the context of the prior art and, unless specifically defined, will not be interpreted in an idealized or overly formal sense.

[0166] For simplicity of description, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because certain steps can be performed in other orders or simultaneously according to the embodiments of the present invention. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0167] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle window intelligent display method, characterized in that: The vehicle window intelligent display method comprises: Collect voice information, analyze the information of pre-search target and pre-search target features in the voice; Parsing the pre-search target and the pre-search target feature information into information of image elements; The on-board camera collects images around the vehicle; The collected images around the vehicle are displayed on the car window according to the driver's perspective; Segmenting the collected images around the vehicle to generate multiple images of objects to be identified; Scanning the real-time collected image of the surroundings of the vehicle according to the pre-search target and the image elements of the pre-search target features; If an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned, a label is added to the image of the object to be identified displayed corresponding to the vehicle window, indicating that it is the pre-search target.

2. The vehicle window intelligent display method according to claim 1, characterized in that: The collecting of voice information and the analysis of the information of the pre-search target and the pre-search target features in the voice include: Parse sentences with feature information and target information in speech; The parsed features are used to point to the parsed target, and the pre-search target and the features of the pre-search target are determined.

3. The vehicle window intelligent display method according to claim 2, characterized in that: The information of parsing the pre-search target and the pre-search target feature information into image elements includes: According to the pre-searched target described by the voice, a sample image corresponding to the pre-searched target is selected from a preset database; According to the features of the pre-searched target described by the speech, pre-compared image features are marked on the example image; According to the sample image and the marked pre-compared image features, determine the similarity between the image of the object to be identified and the sample image; If the image of the object to be identified and the object in the example image are classified as the same type of object, extracting image features of the image of the object to be identified based on the pre-compared image features; If the similarity between the image features of the to-be-identified object image and the image features of the sample image exceeds a preset threshold, the to-be-identified object image is marked and displayed as a pre-search target.

4. The vehicle window intelligent display method according to claim 3, characterized in that: The step of displaying the collected vehicle surrounding image on the vehicle window according to the driver's perspective includes: According to the segmented and collected images around the vehicle, multiple images of objects to be identified are generated, and the images are processed in layers; The layered processing includes displaying the outline of the image of the object to be identified on the vehicle window; Annotations are added according to the image of the object to be identified displayed corresponding to the vehicle window, and the outline of the image of the object to be identified is displayed in a preset display style.

5. The vehicle window intelligent display method according to claim 4, characterized in that: The step of judging the similarity between the to-be-identified object image and the example image based on the example image and the marked pre-compared image features includes: Assign values ​​to image features according to the degree to which they can be identified; According to the assignment of the pre-search target features, the similarity between the image of the object to be identified and the sample image is determined.

6. The vehicle window intelligent display method according to claim 5, characterized in that: If the image of the object to be identified and the object in the example image are classified as the same type of objects, extracting the image features of the image of the object to be identified based on the pre-compared image features includes: Classify the types of objects in the to-be-recognized object image and the sample image according to the preset first-level image features; According to the preset second-level image features, determining the similarity between the image of the object to be identified and the sample image; Wherein, the second level image features are preset according to the pre-compared image features.

7. A car window intelligent display system, characterized in that: The vehicle window intelligent display system comprises: the target recognition module and the target display module; The target recognition module is used to recognize similar to-be-recognized object images in the vehicle surrounding images according to the pre-searched target analyzed in the speech; The target display module is used to display the vehicle surrounding image on the car window corresponding to the driver's perspective, including adding a label similar to the pre-searched target on the car window display corresponding to the image of the object to be identified according to the pre-searched target analyzed in the voice.

8. A vehicle window intelligent display device, characterized in that: The vehicle window intelligent display device comprises: A voice collection module is used to collect voice information and analyze the information of pre-search targets and pre-search target features in the voice; An image conversion module, used for parsing the pre-search target and the pre-search target feature information into information of image elements; Image acquisition module, used for the vehicle camera to collect images around the vehicle; An image display module is used to display the collected images of the vehicle's surroundings on the vehicle window corresponding to the driver's perspective; An image segmentation module is used to segment the collected images around the vehicle and generate multiple images of objects to be identified; An image scanning module, used to scan the real-time collected image of the vehicle surroundings according to the pre-search target and the image elements of the pre-search target features; The image annotation module is used to add annotations to the image of the object to be identified displayed on the window to indicate that it is the pre-search target if an image of the object to be identified that meets the characteristic image elements of the pre-search target is scanned.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the vehicle window intelligent display method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: include: It stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the vehicle window intelligent display method described in any one of claims 1 to 6.

Images