Intelligent screen control system based on large scenarized model

Abstract

The invention discloses an intelligent screen control system based on a large scenarized model, which belongs to the technical field of intelligent vehicle-mounted devices, and comprises a semantic understanding agent used for identifying user requirements according to user text contents and generating operation requirements; the scenarized interface analysis agent is used for performing analysis according to the screenshot image, identifying an interface menu and interface elements and generating a scenarized operation list; the matching agent is used for matching the operation demand with the scenarized operation list, generating an operation instruction and performing interaction determination on the operation instruction and a user; the simulation interaction agent is used for simulating the operation of a user on a vehicle machine screen according to the operation instruction and converting the simulation operation into an operation sequence which can be executed by a vehicle machine operation system; the multi-mode execution agent is used for sending the operation sequence to the vehicle machine; according to the system, through cooperative work of the multi-agent module, accurate recognition and response to user requirements are achieved, and corresponding vehicle-machine interaction operation is generated.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent vehicle-mounted technology, and specifically relates to an intelligent screen control system based on a scenario-based large model. Background Art

[0002] With the development of in-vehicle intelligent systems, users have an increasing demand for intelligent operations during driving. Traditional car systems often require users to perform multiple manual operations to respond to user needs, and cannot efficiently handle complex scenarios. At the same time, existing systems are mostly limited to the applications that come with the car, and support for third-party applications is relatively limited, which cannot meet the diverse needs of users. Therefore, it is necessary to develop a new intelligent screen control system based on a scenario-based large model to solve existing problems. Summary of the invention

[0003] The purpose of the present invention is to provide an intelligent screen control system based on a scenario-based large model to solve the problem of being unable to accurately identify and respond to user needs.

[0004] To achieve the above object, the present invention provides the following technical solution: a smart screen control system based on a scenario-based large model, comprising: Semantic understanding agent, used to identify user needs based on user text content and generate operation requirements; Scenario-based interface analysis agent, used to analyze screenshots, identify interface menus and interface elements, and generate scenario-based operation lists; Matching agent, which is used to match the operation requirements with the scenario-based operation list, generate operation instructions, and confirm the operation instructions with the user interaction; The simulation interactive agent is used to simulate the user's operation on the vehicle screen according to the operation instructions, and convert the simulated operation into an operation sequence that can be executed by the vehicle operating system; Multi-mode execution agent, used to send operation sequences to the vehicle computer.

[0005] Preferably, the generating of the operation requirement includes: sending the user text content request to the server, obtaining the user intention in the current scenario, integrating local and cloud data, and generating preliminary operation requirements.

[0006] Preferably, the identification interface menu includes: a first-level menu, a second-level menu and a third-level menu of the interface; The identification interface elements include: text, content, pictures, icons and buttons.

[0007] Preferably, the matching of the operation requirements and the scenario-based operation list includes: determining specific interface content that meets the user requirements and generating user reply text.

[0008] Preferably, the interactive confirmation of the operation instruction with the user includes: converting the reply text into voice for broadcasting, and interactively confirming the operation content with the user.

[0009] Preferably, simulating the user's operation on the vehicle screen according to the operation instruction includes: matching the interface content of the simulated click, slide or zoom gesture.

[0010] Preferably, the vehicle computer is used to receive and capture the user's voice signal and convert the voice recognition signal into text content.

[0011] Preferably, the server includes: a scene suggestion library, a user preference library, a user library, a multi-mode operation library, and an interface element library; The server is connected to the semantic understanding agent, the scenario interface analysis agent and the simulation interaction agent.

[0012] Technical effects and advantages of the present invention: The intelligent screen control system based on the scenario-based large model is used for intelligent vehicle terminals. Through the collaborative work of multiple intelligent agent modules, it can accurately identify and respond to user needs and generate corresponding vehicle-machine interactive operations. The specific advantages are as follows: 1. Support intelligent control of non-system applications: enrich user usage scenarios and support intelligent control of various third-party applications; 2. Accurate semantic understanding: Through semantic understanding agents, the accuracy of parsing user needs is improved. The language generalization ability is improved, which is more in line with the user's real words; 3. Personalized scene recommendation: Provide personalized scene recommendations based on the user preference database to improve user experience; 4. Multi-modal interaction support: supports multiple gestures and actions to enhance the intelligence and convenience of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the work flow structure of the present invention; Figure 2 Schematic diagram of the hardware position of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Example 1

[0016] The present invention provides Figure 1The intelligent screen control system based on the scenario-based large model shown in the figure includes: a vehicle computer: installed on the center console of the cockpit, supporting touch operation and voice input; supporting on-site sound reception to capture user voice commands; Edge computing box: a high-performance computing device used to quickly process user requests and provide real-time responses; 5G communication module: used to communicate with the server to obtain or update scenario data; Server as cloud service: Located in the center of the computer room, the server has a database.

[0017] The edge computing box has built-in: The semantic understanding agent is used to parse user voice commands, identify user needs, determine end-user needs based on cockpit scenario content, process single or multiple needs, and generate preliminary operation instructions. Through the semantic understanding agent, the user's voice commands can be accurately parsed and converted into specific operation requirements, thereby improving the accuracy of understanding user needs: The scenario-based interface analysis agent can capture, classify and analyze the vehicle screen, sort out the first, second and third level menus of the current interface, identify the text, content, pictures, icons, buttons and other elements on the interface, and analyze possible scenario-based operations; through the scenario-based interface analysis agent, it can dynamically identify the content on the vehicle screen, adapt to different operation scenarios, and enhance the system's scenario-based adaptability: Matching agents match user needs with interface content, find specific interface content that meets user needs, output specific operation instructions, interact multiple times to confirm user needs, and modify operation content; through matching agents, quickly match user needs with current interface content, determine specific operation steps, and achieve efficient matching of user needs and interface content; Simulate interactive agents, convert operation instructions into interactive operations executable by the vehicle OS, and simulate multiple gesture operations such as clicking, sliding, and zooming in. Through simulating interactive agents and multi-mode execution agents, it can complete complex screen operations, support multiple gestures and actions, and provide intelligent multi-mode interactive operations. The multi-mode execution agent converts interactive operations into executable interface requests for the vehicle computer, transmits the operation instructions to the vehicle computer for execution, and supports single or multiple operation executions; through the understanding of the application interface layout, it can execute screen operations of multiple non-system applications, not limited to the factory preset function operations, and realize multi-touch interactive operations that support non-system applications: through the combination of local interface image understanding results and cloud data centers, it supports the growing scenario-based service content and realizes support for a growing scenario-based service chain: Each module interacts with each other through data interfaces and communication protocols, and the server side stores and processes data to form a user preference database.

[0018] Glossary of terms in the present invention: Car computer: refers to a device installed in the center console of a car that uses a touch screen and applications to replace traditional manual operation of related equipment in the car.

[0019] Agent: In a computer system, an agent is a software entity with independent operating and decision-making capabilities.

[0020] Semantic understanding: Parse user language or text to understand their intentions and needs.

[0021] Scenario-based: Provide corresponding functions or services based on specific usage scenarios.

[0022] Simulate interaction: simulate the user's operations on the car screen, such as clicking, sliding, and zooming in.

[0023] Car OS: Car operating system refers to the operating system running on the car's equipment.

[0024] Example 2

[0025] like Figure 2 A control method of an intelligent screen control system based on a scenario-based large model is shown, comprising the following steps: Step 1: The user issues a voice command: Step 11: The user issues a voice command through the vehicle microphone, and the vehicle microphone captures the user's voice signal; Step 12: Convert the voice signal into text through voice recognition technology; Step 13: Send the text content to the semantic understanding agent in the edge computing box for analysis; Step 14: While analyzing, the semantic understanding agent sends a request to the cloud server to obtain the user's possible intention in the current scenario; Step 15: Integrate local and cloud data to generate preliminary operation requirements; Step 2: Scenario-based interface analysis The agent analyzes the current screen: Step 21: Take a screenshot of the vehicle computer to capture the image of the current screen; Step 22: The scenario-based interface analysis agent analyzes the screenshot image and identifies the primary, secondary, and tertiary menus of the interface; Step 23: Identify text, icons, buttons and other elements on the interface; Step 24: Analyze the interface content and generate a list of possible scenario-based operations; Step 25: Update the interface content to the cloud scenario-based interface library; Step 3: The matching agent matches user needs with interface content: Step 31: Match the operation requirements generated by the semantic understanding agent with the scenario-based operation list generated by the scenario-based interface analysis agent; Step 32: Find specific interface content that can meet user needs; Step 33: Generate detailed operation instructions; Step 34, generating user reply text to interact with the user; Step 35: Convert the reply text into voice for broadcast; Step 36, confirm the operation content through multiple interactions; Step 4: Simulate the interactive agent to simulate user operations: Step 41: simulate the user's operation on the vehicle screen according to the operation instructions generated by the matching agent; Step 42: Matching interface content of multiple gestures such as simulated click, slide, and zoom; Step 43: convert the simulated operation into an operation sequence executable by the vehicle OS; Step 5: The multi-modal agent performs the following operations: Step 51, sending the operation sequence generated by the simulated interactive agent to an interface request executable by the vehicle computer; Step 52: Transmit the interface request to the vehicle computer for execution through the 5G communication module; Step 53: Support single or multiple operations to complete all interactive operations required by users.

[0026] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent screen control system based on a scenario-based large model, characterized by: include: Semantic understanding agent, used to identify user needs based on user text content and generate operation requirements; Scenario-based interface analysis agent, used to analyze screenshots, identify interface menus and interface elements, and generate scenario-based operation lists; Matching agent, which is used to match the operation requirements with the scenario-based operation list, generate operation instructions, and confirm the operation instructions with the user interaction; The simulation interactive agent is used to simulate the user's operation on the vehicle screen according to the operation instructions, and convert the simulated operation into an operation sequence that can be executed by the vehicle operating system; Multi-mode execution agent, used to send operation sequences to the vehicle computer.

2. The intelligent screen control system based on scenario-based large model according to claim 1, characterized in that: The generating operation requirements includes: sending the user text content request to the server, obtaining the user intention in the current scenario, integrating local and cloud data, and generating preliminary operation requirements.

3. The intelligent screen control system based on scenario-based large model according to claim 1, characterized in that: The identification interface menu includes: a first-level menu, a second-level menu and a third-level menu of the interface; The identification interface elements include: text, content, pictures, icons and buttons.

4. The intelligent screen control system based on scenario-based large model according to claim 1, characterized in that: The matching of the operation requirements and the scenario-based operation list includes: determining specific interface content that meets the user requirements and generating user reply text.

5. The intelligent screen control system based on scenario-based large model according to claim 1 is characterized in that: The interactive confirmation of the operation instruction with the user includes: converting the reply text into voice for broadcasting, and interactively confirming the operation content with the user.

6. The intelligent screen control system based on scenario-based large model according to claim 1, characterized in that: The simulating the user's operation on the vehicle screen according to the operation instruction includes: matching the interface content of the simulated click, slide or zoom gesture.

7. The intelligent screen control system based on the scenario-based large model according to any one of claims 1 to 6, characterized in that: The vehicle computer is used to receive and capture the user's voice signal and convert the voice recognition signal into text content.

8. The intelligent screen control system based on the scenario-based large model according to any one of claims 1 to 6, characterized in that: The server includes: a scene suggestion library, a user preference library, a user library, a multi-mode operation library, and an interface element library; The server is connected to the semantic understanding agent, the scenario interface analysis agent and the simulation interaction agent.

Images