Vehicle interaction methods, devices, vehicles, and storage media

Abstract

This application relates to a vehicle interaction method, device, vehicle, and storage medium. The method includes: receiving a user's voice control command in a visible-to-speak mode; determining the user's actual intent based on the voice control command, and if the actual intent is a control intent for the central control screen, converting the current display content of the vehicle's central control screen into corresponding text content using a preset large model; after voice-reading the text content, controlling the central control screen to perform corresponding control actions based on the user's response to the text content, and after the control actions are completed, voice-reading the updated display content of the central control screen. This solves the problems of not being able to recognize image information, difficulty in accurately representing screen content, and the high cost of requiring third-party application developers to control related interactions.

Description

Technical Field

[0001] This application relates to the fields of image processing and artificial intelligence technology, and in particular to a vehicle interaction method, device, vehicle, and storage medium. Background Technology

[0002] To achieve efficient information exchange between vehicles and infrastructure, improve driving efficiency, and reduce traffic accidents, vehicles are equipped with voice assistants to reduce driver distraction during vehicle use. Drivers can control the central control interface via voice commands, enabling them to speak what they see instantly; what the user sees can be translated into speech in real time.

[0003] The relevant technologies include the following two solutions:

[0004] Option 1: Utilize Android Accessibility Service. This option acquires the text content currently displayed on the screen through accessibility events, registers this text content as hot words with the iFlytek API, and triggers corresponding actions when the user speaks these hot words via the accessibility service's click function. However, this option cannot recognize image information and only supports native Android interfaces; it cannot capture information from H5, JS, or other similar interfaces.

[0005] Option 2: Based on proactive registration by third-party application developers. Deeply customized applications proactively register different texts from different interfaces with the speech recognition engine via a voice interface. This option requires third-party application developers to control text registration and trigger actions such as clicks and swipes, leading to a significant workload for application development and increased costs for car manufacturers.

[0006] In summary, the relevant technologies cannot recognize image information, making it difficult to accurately represent screen content. Furthermore, the related interactions require control by third-party application developers, resulting in high costs, and therefore require improvement. Summary of the Invention

[0007] This application provides a vehicle interaction method, device, vehicle, and storage medium to solve the technical problems in related technologies, such as the inability to recognize image information, difficulty in accurately displaying screen content, and the need for third-party application developers to control related interactions, resulting in high costs.

[0008] The first aspect of this application provides a vehicle interaction method, comprising the following steps: receiving a user's voice control command in a visible-to-speak mode; determining the user's actual intention based on the voice control command, and if the actual intention is a central control screen control intention, converting the current display content of the vehicle's central control screen into corresponding text content using a preset large model; after voice-reading the text content, controlling the central control screen to perform corresponding control actions based on the user's response command to the text content, and after the control actions are completed, voice-reading the updated display content of the central control screen.

[0009] Optionally, in one embodiment of this application, the step of converting the current display content of the vehicle's central control display screen into corresponding text content using a preset large model includes: extracting multiple image features from the current display content, and determining whether each image feature meets a preset output condition; converting the image features that meet the preset output condition into corresponding text, determining the error type of the image features that do not meet the preset output condition, correcting the image features that do not meet the preset output condition based on the error type, and converting the corrected image features into corresponding text, so as to convert the current display content into text content.

[0010] Optionally, in one embodiment of this application, the step of extracting multiple image features from the currently displayed content includes: preprocessing the currently displayed content to obtain image-enhanced display content; and using the preset large model to extract image features from the display content to obtain multiple image features of the display content.

[0011] Optionally, in one embodiment of this application, the step of determining whether each image feature satisfies the preset output condition includes: determining whether each image feature exists in a preset database, wherein image features existing in the preset database satisfy the preset output condition; determining whether image features not existing in the preset database have string errors; if string errors exist, replacing the erroneous string to obtain a first image feature that satisfies the preset output condition; if no string errors exist, performing classifier prediction on the image features not existing in the preset database to obtain corresponding predicted values, and determining whether the predicted values ​​are less than a preset threshold; if less than the preset threshold, determining that the image features not existing in the preset database satisfy the preset output condition; otherwise, using the predicted category corresponding to the predicted value as the corresponding category of the image feature to obtain a second image feature that satisfies the preset output condition.

[0012] Optionally, in one embodiment of this application, determining the user's actual intent based on the voice command includes: extracting at least one keyword from the voice command; matching each keyword with a keyword mapping relationship in a preset voice database one by one, and determining the actual intent based on the mapping relationship.

[0013] Optionally, in one embodiment of this application, the method further includes: acquiring internal video data of the vehicle; acquiring behavioral characteristics of each occupant in the vehicle based on the internal video data, and determining whether there is a visually impaired occupant in the vehicle who meets the preset visual impairment conditions based on the behavioral characteristics; if there is a visually impaired occupant, then controlling the vehicle to enter the "visible and speakable" mode when a preset keyword is detected in the audio data of the visually impaired member.

[0014] A second aspect of this application provides a vehicle interaction device, comprising: a receiving module for receiving a user's voice control command in a visible-to-speak mode; a conversion module for determining the user's actual intention based on the voice control command, and, if the actual intention is a central control screen control intention, converting the current display content of the vehicle's central control screen into corresponding text content using a preset large model; and an interaction module for, after voice-reading the text content, controlling the central control screen to perform corresponding control actions based on the user's response command to the text content, and, after the control actions are completed, voice-reading the updated display content of the central control screen.

[0015] Optionally, in one embodiment of this application, the conversion module includes: a first extraction unit, configured to extract multiple image features from the currently displayed content, and determine whether each image feature meets a preset output condition; and a conversion unit, configured to convert the image features that meet the preset output conditions into corresponding text, determine the error type of the image features that do not meet the preset output conditions, and correct the image features that do not meet the preset output conditions based on the error type, and convert the corrected image features into corresponding text, so as to convert the currently displayed content into text content.

[0016] Optionally, in one embodiment of this application, the extraction unit includes: a preprocessing subunit, used to preprocess the currently displayed content to obtain image-enhanced display content; and an extraction subunit, used to extract image features from the display content using a preset large model to obtain multiple image features of the display content.

[0017] Optionally, in one embodiment of this application, the extraction unit includes: a first judgment subunit, configured to determine whether each image feature exists in a preset database, wherein image features existing in the preset database satisfy the preset output condition; a second judgment subunit, configured to determine whether image features not existing in the preset database have string errors; a replacement subunit, configured to replace the erroneous string if a string error exists, to obtain a first image feature that satisfies the preset output condition; a third judgment subunit, configured to perform classifier prediction on the image features not existing in the preset database if no string error exists, to obtain a corresponding predicted value, and to determine whether the predicted value is less than a preset threshold; and an output subunit, configured to determine that the image features not existing in the preset database satisfy the preset output condition if the value is less than the preset threshold, otherwise, to use the predicted category corresponding to the predicted value as the corresponding category of the image feature, to obtain a second image feature that satisfies the preset output condition.

[0018] Optionally, in one embodiment of this application, the receiving module includes: a second extraction unit, configured to extract at least one keyword from the voice command; and a matching unit, configured to match each keyword with a keyword mapping relationship in a preset voice database, and determine the actual intent based on the mapping relationship.

[0019] Optionally, in one embodiment of this application, it further includes: an acquisition module, used to acquire internal video data of the vehicle; a judgment module, used to acquire behavioral characteristics of each occupant in the vehicle based on the internal video data, and to determine whether there is a visually impaired occupant in the vehicle who meets the preset visual impairment conditions based on the behavioral characteristics; and a control module, used to control the vehicle to enter the "visible and speakable" mode when a preset keyword is detected in the audio data of the visually impaired occupant if the visually impaired occupant is present.

[0020] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle interaction method as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing the computer to perform the vehicle interaction method as described in the above embodiments.

[0022] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, implements the above-described vehicle interaction method.

[0023] This application embodiment, in a "see-and-say" mode, determines the user's actual intent based on voice control commands. When the actual intent is to control the central control screen, a preset large model is used to convert the current display content of the vehicle's central control screen into corresponding text content. After the text content is read aloud, the central control screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control screen is read aloud, empowering the image recognition system of the in-vehicle central control interface. The voice recognition system enables voice selection of images. By introducing a large-scale deep learning model, the accuracy and efficiency of OCR image recognition are significantly improved. Furthermore, it enables full-screen text and image control via voice, which not only helps drivers concentrate but also allows visually impaired passengers to control the system freely. This solves the problems of inability to recognize image information, difficulty in accurately displaying screen content, and the high cost associated with third-party application developers controlling related interactions.

[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 This is a flowchart of a vehicle interaction method provided according to an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of a speech recognition process according to an embodiment of this application;

[0028] Figure 3 This is a schematic diagram illustrating the principle of a vehicle interaction method according to an embodiment of this application;

[0029] Figure 4 This is a schematic diagram illustrating the principle of histogram equalization processing according to an embodiment of this application;

[0030] Figure 5 This is a schematic diagram illustrating the principle of an image feature extraction method according to an embodiment of this application;

[0031] Figure 6 This is a flowchart illustrating image text recognition according to an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of a vehicle interaction device according to an embodiment of this application;

[0033] Figure 8This is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0035] The following description, with reference to the accompanying drawings, describes a vehicle interaction method, device, vehicle, and storage medium according to embodiments of this application. Addressing the issues mentioned in the background art, such as the inability to recognize image information, difficulty in accurately representing screen content, and the high cost associated with third-party application developer control for related interactions, this application provides a vehicle interaction method. In this method, in a "see-and-say" mode, the user's actual intent is determined based on voice control commands. When the actual intent is a control intent for the central control screen, a preset large model is used to convert the current display content of the vehicle's central control screen into corresponding text content. After the text content is read aloud, the central control screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control screen is read aloud, empowering the image recognition system of the in-vehicle central control interface. The voice recognition system enables voice selection of images. By introducing a large-scale deep learning model, the accuracy and efficiency of OCR image recognition are significantly improved. Furthermore, it enables full-interface text and image "see-and-say" voice control, which not only helps drivers concentrate but also facilitates free control for visually impaired passengers. This solves the problems of not being able to recognize image information, making it difficult to accurately display screen content, and requiring third-party application developers to control related interactions, resulting in high costs.

[0036] Specifically, Figure 1 This is a flowchart illustrating a vehicle interaction method provided in an embodiment of this application.

[0037] like Figure 1 As shown, the interaction method of this vehicle includes the following steps:

[0038] In step S101, in the visible-to-speak mode, the user's voice control command is received.

[0039] Understandably, as an innovative interaction method, "what you see is what you can say" can be applied to fields such as smart devices, in-vehicle systems, and smart homes. Its core lies in using voice recognition technology to allow users to directly operate or input commands on visible elements on the screen without manually touching the screen or other physical buttons, or to play visible elements on the screen by voice, so that users can know the actual content on the screen without seeing it.

[0040] In actual implementation, this embodiment of the application can receive the user's voice control command when the vehicle is in the "visible and speakable" mode, and then determine the vehicle's next action based on the voice control command.

[0041] In step S102, the user's actual intention is determined based on the voice control command, and if the actual intention is to control the central control screen, the current display content of the vehicle's central control screen is converted into corresponding text content using a preset large model.

[0042] As one possible approach, embodiments of this application can determine the user's actual intent based on voice control commands, such as voice control of turning on or off in-vehicle functional devices.

[0043] And when the actual intention is to control the central control screen, a large-scale deep learning model is used, combined with steps such as image preprocessing, feature extraction, model training and text recognition, to convert the current display content of the central control screen into corresponding text content.

[0044] This application embodiment utilizes speech recognition technology to receive and parse user voice commands. Through an efficient speech recognition algorithm, speech is accurately converted into text commands, which are then matched with the text recognized by OCR and corresponding operations are executed.

[0045] This application embodiment can also utilize Natural Language Processing (NLP) technology to understand and parse the intent of voice commands. NLP can help the system identify keywords and actions in the commands, thereby determining the specific operation that the user wants to perform.

[0046] The embodiments of this application can also utilize the vehicle system control interface to provide the ability to interact with the vehicle system hardware and software, and execute instructions parsed by OCR and speech recognition.

[0047] like Figure 2 As shown, the speech recognition process may include the following steps:

[0048] After the user activates the AI ​​voice assistant with a wake word and issues a command, the system recognizes the spoken command, analyzes the user's intent using NLP, and then matches the parsed voice command with the feature categories obtained from OCR recognition using vector matching and large-scale semantic understanding. If the match is successful, the corresponding operation is executed through the vehicle system control interface, such as opening an application, selecting a specific menu item, or adjusting settings. This achieves the functionality of speaking when the text and image are visible. If the information is not related to the interface category, other voice commands or casual conversation functions are implemented.

[0049] Optionally, in one embodiment of this application, determining the user's actual intent based on voice commands includes: extracting at least one keyword from the voice commands; matching each keyword with a keyword mapping relationship in a preset voice database one by one, and determining the actual intent based on the mapping relationship.

[0050] In some embodiments, the user's actual intent can be determined by keyword triggering or other methods. The keywords can be preset by the user or preset by those skilled in the art, and the mapping relationship between different keywords and corresponding functions can be determined.

[0051] Optionally, in one embodiment of this application, the current display content of the vehicle's central control display screen is converted into corresponding text content using a preset large model, including: extracting multiple image features from the current display content, and determining whether each image feature meets preset output conditions; converting the image features that meet the preset output conditions into corresponding text, determining the error type of the image features that do not meet the preset output conditions, correcting the image features that do not meet the preset output conditions based on the error type, and converting the corrected image features into corresponding text, so as to convert the current display content into text content.

[0052] This includes extracting multiple image features from the currently displayed content, including: preprocessing the currently displayed content to obtain the image-enhanced displayed content; and using a preset large model to extract image features from the displayed content to obtain multiple image features of the displayed content.

[0053] The process of determining whether each image feature meets the preset output conditions includes: determining whether each image feature exists in a preset database, wherein image features existing in the preset database meet the preset output conditions; determining whether image features not existing in the preset database have string errors; if string errors exist, replacing the erroneous string to obtain a first image feature that meets the preset output conditions; if no string errors exist, performing classifier prediction on the image features not existing in the preset database to obtain corresponding predicted values, and determining whether the predicted values ​​are less than a preset threshold; if less than the preset threshold, determining that the image features not existing in the preset database meet the preset output conditions; otherwise, using the predicted category corresponding to the predicted value as the corresponding category of the image feature to obtain a second image feature that meets the preset output conditions.

[0054] In actual implementation, such as Figure 3 As shown, the embodiments of this application may involve the following structure when performing image-to-text conversion: an image preprocessing module, an image feature extraction module, a text recognition module, and a speech module.

[0055] The image preprocessing module preprocesses the interface content on the central control display screen by performing operations such as grayscale conversion and image enhancement to improve image quality and reduce the impact of background noise on the recognition results.

[0056] The grayscale conversion process can be summarized as follows: A color image is composed of the three primary colors R, G, and B according to certain values. Each component is stored as 8 bits, and storing one color pixel requires 3*8 bits of space. An 8-bit variable has 255 possible values, and a color pixel can have as many as 255*255*255 possible values; the higher the pixel count, the larger the space required. For high-resolution images, if the model directly processes the color interface, the computational load is too large, the training time is too long, and the memory consumption is too high. Grayscale conversion helps reduce data dimensionality, simplifies image processing algorithms, improves the generalization ability of image recognition models, and makes images easier to analyze and recognize. Four common methods for grayscale conversion of color images are the averaging method, the maximum and minimum value method, the component method, and the weighted average method.

[0057] Based on the relationship between the YUV and RGB color space components, the R, G, and B components are weighted to obtain the final pixel value, such as the weighted average method. Different weights are used to extract luminance information from the color channels. Typically, based on the different color sensitivities of the human eye, green receives the highest weight, followed by red, and then blue. This method can basically meet the requirements of most image processing tasks. Equation 1-1 is as follows:

[0058] Y=(R*0.299+G*0.587+B*0.114)(1-1)

[0059] Image enhancement processes can include: highlighting key elements in an image, increasing contrast, brightness, or sharpening, making people or objects more prominent and easier to identify. These processes can be divided into two categories: spatial domain methods and frequency domain methods.

[0060] The spatial domain can be simply understood as the space containing image pixels. The spatial domain method refers to performing various linear or nonlinear operations directly on the image in the spatial domain, that is, the image itself, to enhance the pixel grayscale values ​​of the image.

[0061] The frequency domain method treats the image as a two-dimensional signal in the transform domain and performs signal enhancement based on the two-dimensional Fourier transform.

[0062] Spatial domain methods are further divided into two main categories: point operations and stencil processing. Point operations are processing methods that operate on the neighborhood of a single pixel, including image grayscale transformation, histogram correction, and pseudo-color enhancement techniques. Stencil processing is processing methods that operate on the pixel-wide area, including image smoothing and image sharpening techniques. Among these, histogram equalization is a relatively simple and effective method that changes the image's grayscale by altering its histogram, thereby enhancing image contrast.

[0063] like Figure 4 As shown in the histogram, if gray levels are concentrated in high-gray-level areas, low-gray-level images are difficult to distinguish; conversely, if gray levels are concentrated in low-gray-level areas, high-gray-level images are difficult to distinguish. To make both high and low gray levels easily distinguishable, the best approach is to transform the image so that the probability of gray level distribution is the same. When the image's histogram is uniformly distributed, the image contrast is more pronounced, resulting in a clearer image. This is the purpose of histogram equalization.

[0064] Implementation steps: Calculate the histogram of the image and normalize it to the range [0,1]. Calculate a mapping function that maps the original gray values ​​to new gray values ​​to make the histogram distribution more uniform. Use the obtained mapping function to process the image, replacing the gray value of each pixel with the new mapped value.

[0065] Image Feature Extraction Module: Since computers can only recognize numbers, not images, the image feature extraction process involves processing and analyzing the information contained in an image, and extracting information less susceptible to random interference as image features. This transforms the original image features into a set of features with clear physical or statistical significance, facilitating subsequent image recognition, classification, and retrieval tasks.

[0066] In interface OCR recognition, image feature extraction mainly involves capturing and extracting key information related to text from the image, such as color, texture, shape, edges, corners, lines, regions, etc., and converting them into vectors, symbols, and numerical features.

[0067] Implementation Scheme: In OCR technology, feature extraction methods mainly include the extraction of color features, texture features, shape features, key points and descriptors, and deep learning features. Among them, color features focus on the frequency and spatial distribution of color occurrences; shape features focus on the overall shape of characters and the thickness of strokes; texture features involve detailed information within characters, such as the direction and density of strokes; key points and descriptors focus on key points in the image; and deep learning features focus on higher-level feature representations, which are particularly effective for complex image recognition tasks.

[0068] Among them, feature extraction methods can be such as Figure 5 As shown.

[0069] Taking Convolutional Neural Networks (CNNs) as an example, we will perform text detection based on deep learning.

[0070] CNNs have the advantages of fewer parameters and translation invariance. VGGNet is a type of CNN, consisting of 5 convolutional layers, 3 fully connected layers, and an output layer. The layers are separated by max-pooling, and all hidden activation units use the ReLU (Rectified Linear Unit) function. It extracts image features through convolutional layers, fully connected layers, and the output layer, and uses max-pooling layers for classification, automatically learning feature representations in the image to achieve content localization. These features can better capture detailed information in the image, improving the accuracy of OCR.

[0071] Implementation steps:

[0072] Step S1: Input layer.

[0073] The image is input into the network in the form of a multi-channel two-dimensional matrix.

[0074] Step S2: Convolutional layer.

[0075] In a convolutional layer, the input image is convolved with a set of learnable convolutional kernels (also called filters). The convolutional kernels slide across the input image, and the dot product between the kernel and a local region of the image is calculated to generate a feature map.

[0076] Each convolutional kernel can extract different features from an image, such as edges and textures.

[0077] By stacking convolutional layers, higher-level abstract features of an image can be gradually extracted.

[0078] Step S3: Activation function.

[0079] Apply a non-linear activation function, such as ReLU, to the feature maps of the convolutional layer.

[0080] Activation functions introduce nonlinear transformations, increasing the model's expressive power and enabling the network to learn and simulate more complex feature patterns.

[0081] Step S4: Pooling layer.

[0082] Pooling layers are used to reduce the spatial dimension of feature maps, reduce the number of parameters, and extract the main features of the image.

[0083] The most commonly used pooling operation is max pooling, which takes the maximum value in each local region as the pooling result.

[0084] Pooling operations are translation invariant, making the network robust to small positional changes in the image.

[0085] Step S5: Multi-layer stacking.

[0086] CNNs are typically composed of multiple convolutional layers and pooling layers stacked alternately.

[0087] Each layer increases the number of features, enabling the network to capture more complex feature patterns.

[0088] By stacking layers one after another, CNNs are able to extract increasingly abstract and semantic features from the original image.

[0089] Step S6: Fully connected layer.

[0090] After multiple convolution and pooling operations, the resulting feature map is flattened into a vector and connected to one or more fully connected layers.

[0091] The role of fully connected layers is to associate these features with category labels for classification or regression.

[0092] Step S7: Output layer.

[0093] The output layer typically uses the Softmax activation function to generate predictions.

[0094] The Softmax function transforms the score for each category into a probability distribution, representing the probability that the sample belongs to each category.

[0095] In summary, CNNs, through layers of stacked convolution and pooling operations, can extract higher-level abstract features from images layer by layer, such as facial contours, eyes, noses, animal shapes, sky, mountains, water, and the sun in video interfaces, as well as the types of controls in system control interfaces. At the same time, CNNs have significantly fewer model parameters compared to traditional neural networks, improving computational efficiency and generalization ability.

[0096] Text recognition module:

[0097] When handling OCR (Optical Character Recognition) tasks, ensuring the main thread is not blocked is crucial. Executing time-consuming tasks like OCR on the main thread can degrade the user experience and may even cause application unresponsiveness or vehicle system crashes. Therefore, the recognition task can be executed in the background via an Android Service. Furthermore, integrating an OCR engine (such as the ABBYY OCR SDK, which boasts up to 99% efficiency for recognizing high-quality images) and loading relevant library files allows for comparison of performance differences.

[0098] For content requiring image recognition, datasets are provided, such as video cover data from in-vehicle infotainment system interfaces and all UI images within the vehicle. Using these labeled image datasets, a VGG (Visual Geometry Group) network model is trained based on the Tensorflow deep learning framework to build character and word classifiers. Based on feature similarity, images are mapped to specific categories or labels. Simultaneously, techniques such as transfer learning and incremental learning are employed to enable rapid adaptation to new domains and tasks.

[0099] For example, the text recognition process in the embodiments of this application can be as follows: Figure 6 As shown.

[0100] In step S103, after the text content is read aloud, the central control display screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control display screen is read aloud.

[0101] As one possible implementation, this application embodiment can receive relevant user instructions after the text content of the central control display screen is read out by voice, and control the central control display screen to perform the next action according to the user's relevant instructions, so as to control the content displayed on the central control display screen or adjust the function of the central control display screen. After the control is completed, the above steps are repeated to read out new content on the central control display screen.

[0102] Operation can be performed without the user looking at the central control display screen, improving vehicle driving safety.

[0103] Optionally, in one embodiment of this application, the method further includes: acquiring internal video data of the vehicle; acquiring behavioral characteristics of each occupant in the vehicle based on the internal video data, and determining whether there are visually impaired occupants in the vehicle who meet the preset visual impairment conditions based on the behavioral characteristics; if there are visually impaired occupants, then controlling the vehicle to enter the "visible and speakable" mode when the audio data of the visually impaired member is identified as containing preset keywords.

[0104] In actual implementation, in addition to the user controlling the vehicle to enter the "visible and speakable" mode by means of button commands, voice commands, etc., the embodiments of this application can also obtain the internal video data of the vehicle, and then obtain the behavioral characteristics of each member in the vehicle from the internal video data, so as to determine whether there are visually impaired occupants in the vehicle, such as whether a guide cane is recognized in the vehicle, whether the occupant's behavior is groping behavior, and occupant pupil recognition, etc.

[0105] When there are visually impaired occupants in the vehicle, the embodiments of this application can specifically identify keywords in the speech of visually impaired occupants, and then control the vehicle to enter the "visible and speakable" mode when matching keywords (e.g., words such as "play", "see", "what").

[0106] The vehicle interaction method proposed in this application allows for a "see-and-say" mode. Based on voice control commands, the user's actual intent is determined. When the actual intent is to control the central control screen, a preset large model is used to convert the current display content of the vehicle's central control screen into corresponding text content. After the text content is read aloud, the central control screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control screen is read aloud. This empowers the image recognition system of the in-vehicle central control interface, enabling voice selection of images. By introducing a large-scale deep learning model, the accuracy and efficiency of OCR image recognition are significantly improved. Furthermore, it allows for full-interface text and image interaction via voice, which not only helps drivers concentrate but also facilitates free control for visually impaired passengers. This solves the problems of inability to recognize image information, difficulty in accurately displaying screen content, and the high cost associated with third-party application development control for related interactions.

[0107] Next, the interactive device for a vehicle according to an embodiment of this application is described with reference to the accompanying drawings.

[0108] Figure 7 This is a block diagram of the vehicle's interactive device according to an embodiment of this application.

[0109] like Figure 7 As shown, the vehicle's interactive device 10 includes a receiving module 100, a conversion module 200, and an interactive module 300.

[0110] Specifically, the receiving module 100 is used to receive the user's voice control commands in the visible-to-speak mode.

[0111] The conversion module 200 is used to determine the user's actual intention based on the voice control command, and when the actual intention is the control intention of the central control screen, it uses a preset large model to convert the current display content of the vehicle's central control screen into corresponding text content.

[0112] The interaction module 300 is used to broadcast text content by voice, and then control the central control display screen to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the module broadcasts the updated display content of the central control display screen by voice.

[0113] Optionally, in one embodiment of this application, the conversion module 200 includes a first extraction unit and a conversion unit.

[0114] The first extraction unit is used to extract multiple image features from the currently displayed content and determine whether each image feature meets the preset output conditions.

[0115] The conversion unit is used to convert image features that meet preset output conditions into corresponding text, determine the error type of image features that do not meet preset output conditions, correct the image features that do not meet preset output conditions based on the error type, and convert the corrected image features into corresponding text, so as to convert the currently displayed content into text content.

[0116] Optionally, in one embodiment of this application, the extraction unit includes a preprocessing subunit and an extraction subunit.

[0117] The preprocessing subunit is used to preprocess the currently displayed content to obtain the enhanced display content.

[0118] The extraction sub-unit is used to extract image features from the displayed content using a preset large model, thereby obtaining multiple image features of the displayed content.

[0119] Optionally, in one embodiment of this application, the extraction unit includes: a first judgment subunit, a second judgment subunit, a replacement subunit, a third judgment subunit, and an output subunit.

[0120] The first judgment subunit is used to determine whether each image feature exists in the preset database, wherein the image features that exist in the preset database satisfy the preset output conditions.

[0121] The second judgment subunit is used to determine whether there is a string error in image features that do not exist in the preset database.

[0122] The replacement subunit is used to replace the erroneous string in the event of a string error, thereby obtaining the first image feature that meets the preset output conditions.

[0123] The third judgment subunit is used to perform classifier prediction on image features that do not exist in the preset database in the absence of string errors, obtain the corresponding predicted value, and determine whether the predicted value is less than the preset threshold.

[0124] The output subunit is used to determine if an image feature that does not exist in the preset database satisfies the preset output condition when the value is less than a preset threshold; otherwise, the predicted category corresponding to the predicted value is used as the corresponding category of the image feature to obtain a second image feature that satisfies the preset output condition.

[0125] Optionally, in one embodiment of this application, the receiving module 100 includes a second extraction unit and a matching unit.

[0126] The second extraction unit is used to extract at least one keyword from the voice command.

[0127] The matching unit is used to match each keyword with the keyword mapping relationship in the preset speech database one by one, and determine the actual intention based on the mapping relationship.

[0128] Optionally, in one embodiment of this application, the vehicle's interaction device 10 further includes an acquisition module, a judgment module, and a control module.

[0129] The acquisition module is used to acquire internal video data of the vehicle.

[0130] The judgment module is used to obtain the behavioral characteristics of each passenger in the vehicle based on internal video data, and to determine whether there are visually impaired passengers in the vehicle who meet the preset visual impairment conditions based on the behavioral characteristics.

[0131] The control module is used to control the vehicle to enter the "visible and speakable" mode when a visually impaired passenger is present and the audio data of the visually impaired passenger contains preset keywords.

[0132] It should be noted that the foregoing explanation of the vehicle interaction method embodiment also applies to the vehicle interaction device of this embodiment, and will not be repeated here.

[0133] The vehicle interaction device proposed in this application embodiment can determine the user's actual intention based on voice control commands in a "visible and speakable" mode. When the actual intention is to control the central control screen, a preset large model is used to convert the current display content of the vehicle's central control screen into corresponding text content. After the text content is read aloud, the central control screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control screen is read aloud, empowering the image recognition system of the in-vehicle central control interface. The voice recognition system enables voice selection of images. By introducing a large-scale deep learning model, the accuracy and efficiency of OCR image recognition are significantly improved. Furthermore, it enables full-interface text and image interaction via voice, which not only helps drivers concentrate but also allows visually impaired passengers to control the device freely. This solves the problems of inability to recognize image information, difficulty in accurately displaying screen content, and the high cost associated with third-party application developers controlling related interactions.

[0134] Figure 8 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0135] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0136] When the processor 802 executes the program, it implements the vehicle interaction method provided in the above embodiments.

[0137] Furthermore, the vehicle also includes:

[0138] Communication interface 803 is used for communication between memory 801 and processor 802.

[0139] The memory 801 is used to store computer programs that can run on the processor 802.

[0140] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0141] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0142] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0143] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0144] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle interaction method described above.

[0145] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the vehicle interaction method provided in this embodiment of the invention.

[0146] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0148] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0150] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0151] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0153] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A vehicle interaction method, characterized in that, Includes the following steps: In the "see and speak" mode, it receives the user's voice control commands; Based on the voice control command, the user's actual intention is determined, and if the actual intention is to control the central control screen, the current display content of the vehicle's central control screen is converted into corresponding text content using a preset large model. After the text content is read aloud, the central control display screen is controlled to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the updated display content of the central control display screen is read aloud. The step of converting the current display content of the vehicle's central control screen into corresponding text content using a preset large model includes: extracting multiple image features from the current display content and determining whether each image feature meets preset output conditions; converting the image features that meet the preset output conditions into corresponding text, determining the error type of the image features that do not meet the preset output conditions, correcting the image features that do not meet the preset output conditions based on the error type, and converting the corrected image features into corresponding text, thereby converting the current display content into text content.

2. The method according to claim 1, characterized in that, The extraction of multiple image features from the currently displayed content includes: The currently displayed content is preprocessed to obtain the enhanced display content. The preset large model is used to extract image features from the displayed content to obtain multiple image features of the displayed content.

3. The method according to claim 1, characterized in that, The step of determining whether each image feature meets the preset output conditions includes: Each image feature is determined to exist in a preset database, wherein the image features existing in the preset database satisfy the preset output condition; Determine whether there is a string error in the image features that do not exist in the preset database; If a string error exists, the erroneous string is replaced to obtain the first image feature that satisfies the preset output conditions; If there is no string error, then a classifier is used to predict the image features that do not exist in the preset database to obtain the corresponding predicted value, and it is determined whether the predicted value is less than a preset threshold. If the value is less than the preset threshold, then the image feature that does not exist in the preset database is determined to satisfy the preset output condition; otherwise, the predicted category corresponding to the predicted value is used as the corresponding category of the image feature to obtain a second image feature that satisfies the preset output condition.

4. The method according to claim 1, characterized in that, Determining the user's actual intent based on the voice control command includes: Extract at least one keyword from the voice control command; Each keyword is matched one by one with the keyword mapping relationship in the preset voice database, and the actual intention is determined based on the mapping relationship.

5. The method according to claim 1, characterized in that, Also includes: Acquire the internal video data of the vehicle; Based on the internal video data, the behavioral characteristics of each passenger in the vehicle are obtained, and based on the behavioral characteristics, it is determined whether there are visually impaired passengers in the vehicle who meet the preset visual impairment conditions. If a visually impaired passenger is present, and if a preset keyword is detected in the audio data of the visually impaired passenger, the vehicle is controlled to enter the "visible and speakable" mode.

6. A vehicle interaction device, characterized in that, include: The receiving module is used to receive the user's voice control commands in the visible-to-speak mode; The conversion module is used to determine the user's actual intention based on the voice control command, and when the actual intention is the central control screen control intention, it uses a preset large model to convert the current display content of the vehicle's central control screen into corresponding text content. The interaction module is used to read the text content aloud by voice, and control the central control display screen to perform corresponding control actions based on the user's response to the text content. After the control actions are completed, the module reads the updated display content of the central control display screen by voice. The conversion module includes: an extraction unit, used to extract multiple image features from the currently displayed content and determine whether each image feature meets a preset output condition; and a conversion unit, used to convert the image features that meet the preset output conditions into corresponding text, determine the error type of the image features that do not meet the preset output conditions, correct the image features that do not meet the preset output conditions based on the error type, and convert the corrected image features into corresponding text, so as to convert the currently displayed content into text content.

7. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle interaction method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the vehicle interaction method as described in any one of claims 1-5.

Images