Vehicle machine control method, device, equipment and medium

Abstract

The invention discloses a vehicle machine control method and device, equipment and a medium, and the method comprises the steps: obtaining user voice data, and carrying out the voice recognition of the user voice data, and obtaining a voice recognition result; when a first valid character corresponding to the user voice data is recognized, performing screenshot operation on the current page to obtain a screenshot picture; generating an operation instruction based on the screenshot picture and the voice recognition result; and performing simulation operation on the vehicle machine according to the operation instruction. According to the method, when the first valid character is recognized, the current page is subjected to screenshot operation, and by controlling the screenshot opportunity of the picture, the correct rate that the picture can be read when being seen is ensured.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and specifically to a vehicle control method, device, equipment, and medium. Background Technology

[0002] Currently, in the "voice-enabled" functionality of connected vehicle applications, OCR (Optical Character Recognition)-based solutions can address specific scenarios: when a user's third-party application is not compatible with the "voice-enabled" solution, or when traditional content recognition fails, the system uses an OCR process to take a screenshot of the current page, uploads it to the cloud for text recognition, and returns the result to the vehicle. However, if the image upload timing is inaccurate, the recognized content will not match what the user actually sees, impacting the user experience. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the present invention provides a vehicle control method, device, equipment and medium to solve at least one defect in the prior art.

[0004] This invention provides a vehicle infotainment system control method, the vehicle infotainment system control method comprising:

[0005] Acquire user voice data and perform speech recognition on the user voice data to obtain speech recognition results;

[0006] When the first valid character corresponding to the user's voice data is recognized, a screenshot is taken of the current page to obtain a screenshot image;

[0007] An operation command is generated based on the screenshot and the speech recognition result;

[0008] The vehicle system is operated in a simulated manner according to the operation instructions.

[0009] In one embodiment of the present invention, the method further includes:

[0010] The screenshot image is transmitted to the server so that the server can generate operation instructions based on the screenshot image and the speech recognition result.

[0011] In one embodiment of the present invention, the step of generating operation instructions based on the screenshot image and the speech recognition result includes:

[0012] The speech recognition result and the screenshot image are used as inputs to a multi-modal large model. The multi-modal large model generates operation instructions based on the position of the region corresponding to the speech recognition result in the screenshot image and the position of the region corresponding to the speech recognition result in the screenshot image, as well as the speech recognition result.

[0013] In one embodiment of the present invention, before uploading the screenshot image to the server, the method further includes:

[0014] The screenshot image is optimized to obtain an optimized image. The optimization process includes at least one of the following: quality compression and size compression.

[0015] In one embodiment of the present invention, the method includes: converting the optimized image into a format to obtain a base64 representation of the optimized image.

[0016] In one embodiment of the present invention, if there are multiple screenshot operations on the current page, the screenshot images obtained from two adjacent screenshot operations are compared.

[0017] If two consecutive screenshot operations produce the same screenshot image, then the screenshot image from the previous screenshot operation will be transmitted to the server.

[0018] In one embodiment of the present invention, the simulated operation includes at least one of the following: a click operation and a swipe operation.

[0019] The present invention provides a vehicle infotainment control device, the vehicle infotainment control device comprising:

[0020] A speech recognition module is used to acquire user speech data and perform speech recognition on the user speech data to obtain speech recognition results;

[0021] The image capture module is used to capture the current page and obtain a screenshot image when the first valid character corresponding to the user's voice data is recognized.

[0022] The location determination module is used to generate operation instructions based on the screenshot image and the speech recognition result;

[0023] The vehicle control module is used to simulate operations on the vehicle system according to the operation instructions.

[0024] The present invention provides a vehicle control device, comprising:

[0025] One or more processors; and

[0026] A memory for storing one or more programs, which, when executed by one or more processors, enable the vehicle control method described above.

[0027] The present invention provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause the processors to perform the vehicle control method.

[0028] The beneficial effects of this invention are:

[0029] This invention discloses a vehicle infotainment system control method, comprising: acquiring user voice data and performing voice recognition on the user voice data to obtain a voice recognition result; taking a screenshot of the current page when the first valid character corresponding to the user voice data is recognized, thereby obtaining a screenshot image; generating an operation command based on the screenshot image and the voice recognition result; and performing simulated operation on the vehicle infotainment system according to the operation command. This invention performs a screenshot of the current page when the first valid character is recognized, and by controlling the timing of the screenshot, it ensures the accuracy of "what you see is what you can say." Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0031] In the attached diagram:

[0032] Figure 1 This is a flowchart of a vehicle control method according to an embodiment of the present invention;

[0033] Figure 2 This is a block diagram of a vehicle control device according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a computer system suitable for implementing an embodiment of the present invention is shown. Detailed Implementation

[0035] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0036] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0038] The embodiments of the present invention provide a vehicle control method, a vehicle control method apparatus, a vehicle control device, and a computer-readable storage medium, which will be described in detail below.

[0039] Please see Figure 1 , Figure 1 This is a flowchart of a vehicle control method according to an embodiment of the present invention. Figure 1 As shown, the vehicle control method includes steps S110-S140:

[0040] Step S110: Obtain user voice data and perform speech recognition on the user voice data to obtain speech recognition results;

[0041] Step S120: When the first valid character corresponding to the user's voice data is recognized, a screenshot is taken of the current page to obtain a screenshot image;

[0042] Step S130: Generate operation instructions based on the screenshot image and speech recognition results;

[0043] Step S140: Perform simulated operation on the vehicle system according to the operation instructions.

[0044] Specifically, during the user's voice command issuance, the system continuously detects the starting point of a valid command through real-time voice stream analysis. When the first meaningful word is detected, such as recognizing "turn on" as the trigger node in "please turn on the air conditioner," the system immediately calls the screenshot interface to obtain a complete image of the currently displayed page. This process avoids the time lag of waiting for the sentence to end before taking a screenshot, as is done in traditional solutions. Then, the obtained real-time screenshot and voice recognition results are used to generate operation commands, which are directly applied to the vehicle's infotainment system to enable operation.

[0045] It's worth noting that screenshots of the current page can be taken using the SurfaceControl method. Using SurfaceControl for screenshots avoids saving the data to a local file; the screenshot data is obtained only within memory, reducing I / O operations and significantly improving screenshot efficiency.

[0046] When a user triggers the "Speak When You See" feature, the system immediately uses the SurfaceControl API (Application Programming Interface) to capture the screen content and stores the screenshot data directly in a Bitmap object in memory, instead of writing it to the SD card or internal storage. This avoids file system I / O operations and significantly reduces the time required to take a screenshot.

[0047] Through the above technical solution, the present invention effectively solves the problem of recognition error caused by inaccurate screenshot timing.

[0048] In one embodiment, the method further includes: transmitting the screenshot image to a server so that the server can generate operation instructions based on the screenshot image and the speech recognition results.

[0049] Specifically, after the vehicle takes a screenshot of the current page, the screenshot is sent to the cloud server via a network transmission channel. This can be achieved by establishing a real-time communication link using HTTP or WebSocket protocols. The server processes the received screenshot and the speech recognition results to generate an operation command, which is then sent back to the vehicle for execution.

[0050] The server can be a server that provides various services. It can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This section does not impose any restrictions on this.

[0051] This invention reduces the consumption of vehicle-side computing resources by processing multimodal data uniformly on the server side.

[0052] In one embodiment, generating operation instructions based on screenshot images and speech recognition results includes:

[0053] The speech recognition result and the screenshot image are used as inputs to the multi-modal large model. The multi-modal large model generates operation instructions based on the position of the region corresponding to the speech recognition result in the screenshot image and the position of the region corresponding to the speech recognition result in the screenshot image and the speech recognition result.

[0054] The multi-modal large-scale model is a deep learning model capable of simultaneously processing speech, text, and image data. It can employ models such as the GPT-4V large-scale model, the Gemini 1.5 large-scale model, or the Claude 3 large-scale model. Specifically, when a user utters the voice command "adjust the temperature," the system immediately triggers a screenshot upon detecting "adjust" as a valid starting word. At this point, the speech recognition result and the real-time screenshot are simultaneously input into the multi-modal large-scale model. This model determines "temperature" as the operation object through semantic parsing and simultaneously locates the coordinates of the region corresponding to the speech recognition result (temperature adjustment control) in the image feature space. It then maps the spatial relationship between the speech recognition result and the position information of the temperature adjustment control, and combines this with the intent parsing of the speech recognition result to generate a vehicle-mounted operation command containing parameters such as coordinate points and operation type (click operation, swipe operation), and sends this command back to the vehicle for execution.

[0055] It's important to note that the cloud always uses the latest images to perform inference when obtaining speech recognition results. This improves the consistency between the recognition results and the current interface state. This method effectively solves the recognition error problem caused by user operations or system updates that change the interface, improving the accuracy and reliability of the "see it, speak it" function.

[0056] For example, a user says the voice command "Open music." The system begins recognizing the speech, continuously acquiring and uploading the latest screenshots. Once speech recognition is complete, the cloud doesn't use the initially uploaded screenshot; instead, it uses the most recently uploaded screenshot to perform inference. This way, even if the interface changes during speech recognition (e.g., the user manually switches pages), the system can still perform operations based on the latest screenshot (interface state), avoiding operational errors.

[0057] In one embodiment, before uploading the screenshot to the server, the method further includes: optimizing the screenshot to obtain an optimized image, wherein the optimization includes at least one of the following: quality compression and size compression.

[0058] Quality compression refers to reducing image resolution by discarding visual information that is relatively insensitive to the human eye (such as high-frequency details and subtle color changes) while maintaining stable color attributes. Different compression rates are applied to different target areas during quality compression; for example, a lower compression rate is used for text and chart areas to preserve details, while a higher compression rate is used for background areas to reduce data volume.

[0059] Size compression reduces the total number of pixels in an image.

[0060] Specifically, the Android system's bitmap operation interface can be used to compress the quality and size of screenshot images. By performing dual compression of the quality and size of screenshot images, the amount of I / O data in network transmission can be reduced while ensuring image quality.

[0061] It should be noted that when optimizing screenshots, the complexity of the image content needs to be determined first. Complexity detection can be achieved by integrating OpenCV's image complexity analysis module. Then, the order of size compression and quality compression is determined based on the image complexity. If the complexity is greater than or equal to a set threshold, quality compression is performed first, followed by size compression. If the complexity is less than the set threshold, size compression is performed first, followed by quality compression.

[0062] It should be noted that during the size compression process, the system records the scaling ratio of the size compression for subsequent coordinate restoration.

[0063] In one embodiment, the method includes: converting the optimized image to a base64 representation of the optimized image.

[0064] Specifically, the image is converted from binary encoding to a text encoding format based on ASCII characters. This can be achieved using the base64 encoding algorithm to complete the image format conversion. Base64 represents a string data format consisting of 64 printable characters. This can be implemented using the encoding rules defined in RFC 4648. By mapping the original image data to AZ, az, 0-9, and "+" and " / " symbol sequences, it ensures that byte alignment information is not lost during text protocol transmission.

[0065] In one embodiment, if multiple screenshot operations are performed on the current page, the screenshot images obtained from two adjacent screenshot operations are compared; if the screenshot images obtained from two adjacent screenshot operations are the same, the screenshot image obtained from the previous screenshot operation is transmitted to the server.

[0066] Among them, comparing whether the screenshots obtained from two adjacent screenshot operations are the same can be done by using hash algorithms or pixel comparison techniques to determine image similarity. Specifically, this can be achieved by calculating image feature values ​​and setting a similarity threshold.

[0067] Specifically, when multiple screenshots are detected consecutively by a user, the feature fingerprints of adjacent images are extracted using an image hashing algorithm and compared. If the hash values ​​match, it is determined to be a duplicate screenshot. In this case, only the first screenshot is uploaded to the cloud server to avoid repeated transmission of the same content.

[0068] After the operation coordinates are returned from the cloud, the coordinate points are scaled back to the original screen size according to the image scaling parameters recorded locally. For example, for an image that has been compressed by 50%, the coordinate values ​​returned from the cloud are multiplied by 2 to restore the original interface position.

[0069] This invention automatically filters out duplicate screenshots, allowing the cloud server to process only valid image data, thus reducing data transmission volume. At the same time, it ensures that the coordinates of the compressed image are accurately mapped to the actual screen position of the vehicle, avoiding click deviations caused by changes in image size and improving the response accuracy of the voice control function.

[0070] In one embodiment, the simulated operation includes at least one of the following: a click operation and a swipe operation.

[0071] Among them, the click operation triggers discrete interactive behaviors of interface controls through coordinate positioning. This can be achieved by obtaining the center point coordinates of the target area and sending touch event commands to the vehicle system.

[0072] Among them, the sliding operation achieves continuous interactive behavior by simulating the trajectory of the starting and ending coordinates.

[0073] It should be noted that after obtaining the coordinates, the fastest execution method (such as runtime process via shell commands or instrumentation interface) is selected to perform the click or swipe operation.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0075] Figure 2 This is a block diagram of a vehicle control method apparatus according to an embodiment of the present invention. Figure 2 As shown, a vehicle control method apparatus includes:

[0076] The speech recognition module 210 is used to acquire user speech data and perform speech recognition on the user speech data to obtain speech recognition results;

[0077] Image capture module 220 is used to capture the current page when the first valid character corresponding to the user's voice data is recognized, and to obtain a screenshot image.

[0078] Location determination module 230 is used to generate operation instructions based on the screenshot image and the speech recognition result;

[0079] The vehicle control module 240 is used to simulate the operation of the vehicle system according to the operation instructions.

[0080] It should be noted that the vehicle control method and apparatus provided in the above embodiments belong to the same concept as the vehicle control method provided in the above embodiments. The specific ways in which each module and unit performs its operation have been described in detail in the method embodiments and will not be repeated here. In practical applications, the vehicle control method and apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0081] Embodiments of the present invention also provide a vehicle control device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the memory implements the vehicle control method in the above embodiments.

[0082] Embodiments of the present invention also provide one or more machine-readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the vehicle control method described in the above embodiments.

[0083] Figure 3 A schematic diagram of a computer system suitable for implementing an embodiment of the present invention is shown. It should be noted that... Figure 3 The computer system with the memory shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0084] like Figure 3 As shown, the computer system 300 includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage into Random Access Memory (RAM) 303. The RAM also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0085] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0086] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for performing the vehicle control method of the aforementioned embodiments. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, it performs various functions defined in the system of the present invention.

[0087] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM) 303, read-only memory (ROM) 302, erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0089] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0090] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the aforementioned vehicle control method. This computer-readable storage medium may be included in the memory described in the above embodiments, or it may exist independently and not incorporated into that memory.

[0091] Another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the vehicle control methods provided in the various embodiments described above.

[0092] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A vehicle control method, characterized in that, The vehicle control method includes: Acquire user voice data and perform speech recognition on the user voice data to obtain speech recognition results; When the first valid character corresponding to the user's voice data is recognized, a screenshot is taken of the current page to obtain a screenshot image; An operation command is generated based on the screenshot and the speech recognition result; The vehicle system is operated in a simulated manner according to the operation instructions.

2. The vehicle control method according to claim 1, characterized in that, The method further includes: The screenshot image is transmitted to the server so that the server can generate operation instructions based on the screenshot image and the speech recognition result.

3. The vehicle control method according to claim 1 or 2, characterized in that, The step of generating operation instructions based on the screenshot image and the speech recognition result includes: The speech recognition result and the screenshot image are used as inputs to a multi-modal large model. The multi-modal large model generates operation instructions based on the position of the region corresponding to the speech recognition result in the screenshot image and the position of the region corresponding to the speech recognition result in the screenshot image, as well as the speech recognition result.

4. The vehicle control method according to claim 2, characterized in that, Before uploading the screenshot to the server, the method further includes: The screenshot image is optimized to obtain an optimized image. The optimization process includes at least one of the following: quality compression and size compression.

5. The vehicle control method according to claim 4, characterized in that, The method includes: The optimized image is then converted to its base64 representation.

6. The vehicle control method according to claim 2, characterized in that, If multiple screenshot operations are performed on the current page, the screenshot images obtained from two adjacent screenshot operations are compared. If two consecutive screenshot operations produce the same screenshot image, then the screenshot image from the previous screenshot operation will be transmitted to the server.

7. The vehicle control method according to claim 1, characterized in that, The simulated operation includes at least one of the following: click operation, swipe operation.

8. A vehicle control device, characterized in that, The vehicle control device includes: A speech recognition module is used to acquire user speech data and perform speech recognition on the user speech data to obtain speech recognition results; The image capture module is used to capture the current page and obtain a screenshot image when the first valid character corresponding to the user's voice data is recognized. The location determination module is used to generate operation instructions based on the screenshot image and the speech recognition result. The vehicle control module is used to simulate operations on the vehicle system according to the operation instructions.

9. A vehicle control device, characterized in that, include: One or more processors; and A memory for storing one or more programs, which, when executed by one or more processors, enable the vehicle control method as described in any one of claims 1-7.

10. A machine-readable medium, characterized in that, It stores instructions that, when executed by one or more processors, cause the processors to perform the vehicle control method as described in any one of claims 1-7.

Images