Method, device and equipment for human-computer interaction in projection display and storage medium

Abstract

The present disclosure provides a human-computer interaction method and device in projection display, equipment and storage medium, relates to the technical field of artificial intelligence, specifically the technical field of voice technology, Internet of Vehicles, human-computer interaction, intelligent driving and the like, and can be applied to automatic driving, unmanned driving and the like. The human-computer interaction method in projection display comprises: receiving a voice instruction, the voice instruction instructing a target operation on a target page element; in response to the voice instruction, searching for the target page element in a projection page element, the projection page element being a page element displayed on a vehicle terminal by a user terminal; in the case where the projection page element comprises the target page element, controlling the user terminal to execute the target operation on the target page element according to attribute information of the target page element. Thus, the "visible means sayable" in projection display is realized, the convenience of human-computer interaction is improved, and the safety of human-computer interaction in a driving scene is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of voice technology, vehicle networking technology, human-computer interaction, and intelligent driving, and can be applied to scenarios such as autonomous driving and driverless driving. In particular, it relates to a human-computer interaction method, device, equipment, and storage medium in projection display. Background Technology

[0002] In vehicle scenarios, vehicle terminals can interconnect with user terminals and project the user terminal's display screen to enrich the vehicle terminal's ecosystem.

[0003] In related technologies, after the user terminal's display screen is projected onto the vehicle terminal, if the user wants to open or control the application on the user terminal, they need to perform manual operations such as screen tapping or screen swiping on the vehicle terminal.

[0004] However, manually operating the in-vehicle terminal to control applications on the user terminal while the vehicle is in motion poses a significant security risk. Summary of the Invention

[0005] This disclosure provides a human-computer interaction method, apparatus, device, and storage medium in a projection display for improving human-computer interaction safety in driving scenarios.

[0006] According to a first aspect of this disclosure, a human-computer interaction method in a projection display is provided, comprising:

[0007] Receive a voice command, which instructs to perform a target operation on a target page element;

[0008] In response to the voice command, the target page element is searched among the projected page elements, where the projected page element is the page element that is projected and displayed on the vehicle terminal by the user terminal.

[0009] When the projected page element includes the target page element, the user terminal is controlled to perform the target operation on the target page element according to the attribute information of the target page element.

[0010] According to a second aspect of this disclosure, a human-computer interaction device for projection display is provided, comprising:

[0011] A receiving unit is configured to receive a voice command, which instructs a target operation to be performed on a target page element.

[0012] The search unit is used to search for the target page element in response to the voice command, wherein the projected page element is a page element that is projected and displayed on the vehicle terminal by the user terminal.

[0013] The control unit is configured to, when the projected page element includes the target page element, control the user terminal to perform the target operation on the target page element according to the attribute information of the target page element.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a human-computer interaction method in a projection display as described in the first aspect.

[0015] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a human-computer interaction method in a projection display as described in the first aspect.

[0016] According to a fifth aspect of this disclosure, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program causing the electronic device to perform a human-computer interaction method in a projection display as described in the first aspect.

[0017] According to a fifth aspect of this disclosure, an intelligent driving vehicle is provided, including electronic devices as described in the third aspect.

[0018] According to the technical solution provided in this disclosure, in response to a voice command instructing a target operation on a target page element, the target page element is searched among the page elements projected onto the user terminal and displayed on the vehicle terminal. If the target page element is found among the page elements projected onto the user terminal and displayed on the vehicle terminal, the user terminal is controlled to perform the target operation on the target page element based on its attribute information. This achieves "what you see is what you speak" in the projection display of the vehicle terminal, allowing users to operate the page elements projected onto the vehicle terminal via voice commands without requiring manual operation on the user terminal or the vehicle terminal. This improves the convenience of human-computer interaction, enhances the safety of human-computer interaction in driving scenarios, and increases the vehicle's intelligence level.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic diagram illustrating an application scenario to which this disclosure applies;

[0022] Figure 2 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 1 ;

[0023] Figure 3 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 2 ;

[0024] Figure 4 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 3 ;

[0025] Figure 5 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 4 ;

[0026] Figure 6 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 5 ;

[0027] Figure 7 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 6 ;

[0028] Figure 8 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 7 ;

[0029] Figure 9 Schematic diagram of the structure of the human-computer interaction device in the projection display provided in the embodiments of this disclosure. Figure 1 ;

[0030] Figure 10 Schematic diagram of the structure of the human-computer interaction device in the projection display provided in the embodiments of this disclosure. Figure 2 ;

[0031] Figure 11 This is a schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure. Detailed Implementation

[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] In in-vehicle scenarios, if users want to operate applications projected from their phones onto the car's infotainment system—such as opening an application, clicking a button within an application, or swiping a page—they typically perform these actions on either the car's screen or their phone's screen. This human-computer interaction method is not very convenient and poses certain safety hazards while the vehicle is in motion.

[0034] To address the aforementioned shortcomings, this disclosure provides a human-computer interaction method, apparatus, device, and storage medium for projection displays, applicable to the field of artificial intelligence technology, specifically voice technology, vehicle networking technology, human-computer interaction, and intelligent driving, and applicable to scenarios such as autonomous driving and driverless driving. In the human-computer interaction method for projection displays, a voice command is received, instructing a target operation on a target page element. When the target page element is located within the page elements projected onto the in-vehicle terminal from the user terminal, the user terminal is controlled to perform the target operation on the target page element based on its attribute information. This achieves "what you see is what you can say" in projection displays, improving the convenience of human-computer interaction, driving safety, and the intelligence level of the vehicle.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario applicable to an embodiment of this disclosure. In this application scenario, the devices involved include a vehicle 110 and a user terminal 120. The vehicle 110 is equipped with an in-vehicle terminal 111, which may have a voice application (also known as a voice assistant) installed, enabling voice interaction. The user terminal 120 can communicate with the in-vehicle terminal 111 wirelessly or via a wired connection, projecting its own display onto the display screen of the in-vehicle terminal 111. During human-computer interaction, the in-vehicle terminal 111 can receive user voice and, based on the user's voice, control the user terminal 120 to operate the page elements projected onto the in-vehicle terminal 111.

[0036] Optional, such as Figure 1 As shown, a communication interconnection application can be installed on the vehicle terminal 111, and a communication interconnection application can also be installed on the user terminal 120. Figure 1(The communication interconnection application on the user terminal 120 is not shown in the figure). The vehicle terminal 111 and the user terminal 120 communicate wirelessly with each other through their respective communication interconnection applications.

[0037] Among them, the user terminal can be a personal digital assistant (PDA) device, a handheld device with wireless communication function (such as a smartphone or tablet), a computing device (such as a personal computer or PC), a wearable device (such as a smartwatch or smart bracelet), and a smart home device (such as a smart speaker or smart display device).

[0038] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0039] Figure 2 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 1 .like Figure 2 As shown, the human-computer interaction methods in projection display include:

[0040] S201, Receive voice command, the voice command instructs to perform the target operation on the target page element.

[0041] The target page element can be an interactive layout element on the application page, or it can be the application's icon. For example, when a user uses voice commands to click on layout elements such as input boxes or buttons on an application page, the target page element is the input box or button. Similarly, when a user uses voice commands to open or close an application, the target page element is the application's icon. There can be one or more target page elements. When a user interacts with multiple page elements via voice, there are multiple target page elements.

[0042] The target operation may include one or more interactive operations. For example, the target operation may include one or more interactive operations such as click, input, long press, swipe, switch, start, close, and exit.

[0043] In this embodiment, the in-vehicle terminal can receive user voice commands when the voice application is active. For example, after waking up the voice application on the in-vehicle terminal, the user speaks a voice command instructing the user to enter the destination in the navigation application, and the in-vehicle terminal receives the voice command.

[0044] S202, in response to a voice command, searches for the target page element among the projected page elements, which are the page elements projected onto the vehicle terminal by the user terminal.

[0045] After establishing a communication connection between the user terminal and the vehicle-mounted terminal, the user terminal can send its own display content to the vehicle-mounted terminal. The vehicle-mounted terminal then displays a projected image based on the user terminal's display content. This projected image contains multiple page elements, all of which are page elements from the user terminal's display content—that is, the page elements projected from the user terminal onto the vehicle-mounted terminal. For simplicity and ease of distinction, these page elements in the projected image are referred to as projected page elements.

[0046] In this embodiment, after receiving a voice command, the vehicle-mounted terminal can perform voice recognition on the command to obtain a voice recognition result. Based on the voice recognition result, it is determined that the voice command instructs a target operation on a target page element. Then, the target page element can be searched among the projected page elements to determine whether the projected page elements include the target page element, i.e., whether the target page element belongs to the page elements projected onto the vehicle-mounted terminal by the user terminal.

[0047] S203, when the projected page elements include the target page elements, control the user terminal to perform the target operation on the target page elements according to the attribute information of the target page elements.

[0048] The attribute information of the target page element may include at least one of the following: the target page element's identification information, the target page element's position information, and the interactive operations supported by the target page element.

[0049] In this embodiment, the vehicle-mounted terminal can obtain attribute information of the projected page elements through the communication connection between itself and the user terminal. If the projected page elements include a target page element, the vehicle-mounted terminal can obtain the attribute information of the target page element from the attribute information of the projected page elements, and send the target page element's attribute information and the target operation to the user terminal, so that the user terminal can perform the target operation on the target page element according to its attribute information.

[0050] In this embodiment of the disclosure, in response to a voice command, the target page element indicated by the voice command is located among the projected page elements. If the projected page elements include the target page element, the user terminal is controlled to perform the target operation on the target page element based on its attribute information. This enables voice interaction with projected page elements, achieving "what you see is what you can say" in projection display scenarios. This improves the convenience of human-computer interaction with projected page elements, reduces safety hazards associated with human-computer interaction while driving, and enhances the vehicle's intelligence level.

[0051] Figure 3 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 2 .like Figure 3 As shown, the human-computer interaction methods in projection display include:

[0052] S301 receives voice commands, which instruct the target page element to perform the target operation.

[0053] The implementation principle and technical effect of S301 can be referred to the aforementioned embodiments, and will not be repeated here.

[0054] S302, in response to a voice command, determines whether the target application to which the target page element belongs is an application on the user terminal or an application on the vehicle terminal.

[0055] In this context, "application on user terminal" refers to the application installed on the user terminal, while "application on vehicle terminal" refers to the application installed on the vehicle terminal. For projected page elements, the application to which the projected page element belongs is the application installed on the user terminal.

[0056] In this embodiment, after receiving a voice command and determining that the voice command instructs a target operation on a target page element, the vehicle terminal can determine the application to which the target page element belongs. For simplicity and ease of distinction, the application to which the target page element belongs is referred to as the target application. Next, the vehicle terminal can determine whether the target application is an application on the user terminal or an application on the vehicle terminal. Any of the following determination results may occur: the target application is an application on the vehicle terminal and not an application on the user terminal; the target application is an application on the vehicle terminal and an application on the user terminal; or the target application is an application on the user terminal and not an application on the vehicle terminal. If the target application is an application on the user terminal and not an application on the vehicle terminal, then S303 can continue to be executed. If the target application is an application on the vehicle terminal and an application on the user terminal, or if the target application is an application on the vehicle terminal and not an application on the user terminal, then no further steps need to be executed.

[0057] Optionally, if the target application is an application on both the vehicle terminal and the user terminal, or if the target application is an application on the vehicle terminal but not the user terminal, then the vehicle terminal performs the target operation on the target page element; or, the vehicle terminal may prompt the user that the target application is not an application on the user terminal.

[0058] S303, if the target application is an application on the user terminal and not an application on the vehicle terminal, then look for the target page element in the projected page elements.

[0059] S304, when the projected page elements include the target page elements, control the user terminal to perform the target operation on the target page elements according to the attribute information of the target page elements.

[0060] In this embodiment, when the target application is an application on a user terminal and not an application on an in-vehicle terminal, S303 and S304 are executed. The implementation principles and technical effects of S303 and S304 can be referred to the aforementioned embodiments, and will not be repeated here.

[0061] In this embodiment, it is first determined whether the target application to which the target page element belongs is an application on the user terminal and not an application on the vehicle terminal. If the target application is an application on the user terminal and not an application on the vehicle terminal, it is determined whether the target page element indicated by the voice command is projected onto the display screen of the vehicle terminal. If the target page element is projected onto the display screen of the vehicle terminal, the vehicle terminal, based on its communication connection with the user terminal, controls the user terminal to perform the target operation on the target page element according to the attribute information of the target page element. In this way, voice interaction in projection display is realized, improving the convenience and security of human-computer interaction while also improving the accuracy of voice interaction.

[0062] In some embodiments, in determining whether the target application to which the target page element belongs is an application on a user terminal or an application on an in-vehicle terminal, one possible implementation includes: determining whether the target application is an application on an in-vehicle terminal; if the target application is not an application on an in-vehicle terminal, obtaining a list of applications corresponding to the user terminal; determining whether the application list contains attribute information of the target application; if the application list contains attribute information of the target application, determining that the target application is an application on the user terminal, otherwise determining that the target application is not an application on the user terminal.

[0063] The application list corresponding to the user terminal contains attribute information of multiple applications on the user terminal. The application attribute information may include the application's identification information, which may include the application's icon, application name, and the name of the corresponding program package (such as the name of the installation package or the name of the configuration data package).

[0064] In this embodiment, the identification information of the target application can be matched with the identification information of the application on the vehicle terminal. If the identification information of the target application exists in the identification information of the application on the vehicle terminal, then the target application is determined to be an application on the vehicle terminal; otherwise, the target application is determined not to be an application on the vehicle terminal. If the target application is not an application on the vehicle terminal, then it is determined whether the target application is an application on the user terminal.

[0065] The vehicle-mounted terminal can obtain and store a list of applications corresponding to the user terminal in advance through a communication connection between itself and the user terminal. After determining that the target application is not an application on the vehicle-mounted terminal, it retrieves the list of applications corresponding to the user terminal from the storage space. Alternatively, after determining that the target application is not an application on the vehicle-mounted terminal, the vehicle-mounted terminal can retrieve the list of applications corresponding to the user terminal from the user terminal. Then, the vehicle-mounted terminal can check whether the application list contains the attribute information of the target application, specifically whether it contains the identification information of the target application. If the application list contains the attribute information of the target application, it is determined that the target application is an application on the user terminal; otherwise, it is determined that the target application is not an application on the user terminal. Therefore, based on the application list corresponding to the user terminal, the accuracy and efficiency of determining whether a target application is an application on the user terminal are improved.

[0066] Furthermore, the in-vehicle terminal can obtain a list of applications corresponding to the user terminal from the memory space corresponding to the voice application; the voice application is used to execute voice interaction functions on the in-vehicle terminal. Specifically, the in-vehicle terminal can obtain a list of applications corresponding to the user terminal from the user terminal, store the application list in the memory space corresponding to the voice application, and then, during the process of determining whether the target application is an application on the user terminal, retrieve the application list corresponding to the user terminal from the memory space corresponding to the voice application. This improves the efficiency of obtaining the application list and enhances the efficiency of human-computer interaction in the projection display.

[0067] This approach relies on storing the application manifest in the memory space corresponding to the voice application. Figure 4 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 3 .

[0068] like Figure 4 As shown, the human-computer interaction methods in projection display include:

[0069] S401: After establishing a communication connection between the vehicle terminal and the user terminal, it obtains an application list from the user terminal. The application list contains attribute information of multiple applications on the user terminal.

[0070] In this embodiment, after the vehicle terminal and the user terminal establish a communication connection, the display screen of the user terminal can be projected onto the projection screen of the vehicle terminal. The vehicle terminal can request the application list from the user terminal and receive the application list returned by the user terminal.

[0071] In one possible implementation, after the vehicle-mounted terminal and the user terminal establish a communication connection, the vehicle-mounted terminal can send an application retrieval request to the user terminal; the vehicle-mounted terminal then receives a list of applications returned by the user terminal in response to the application retrieval request. The application retrieval request is used to request attribute information of applications of the target type, and the application list includes attribute information of applications of the target type on the user terminal.

[0072] The target type can include the type of application that the in-vehicle terminal supports for projection display. For example, target types include image playback, audio and video playback, navigation, and map types.

[0073] In this implementation, after establishing a communication connection between the vehicle-mounted terminal and the user terminal, the vehicle-mounted terminal can send an application acquisition request to the user terminal. This request may include a target type, requesting the acquisition of attribute information for applications of the target type. Upon receiving the application acquisition request, the user terminal can respond by retrieving the attribute information of the target type applications from the attribute information of its installed applications and returning this information to the vehicle-mounted terminal. Therefore, by considering the types of applications supported for projection display by the vehicle-mounted terminal during the application list acquisition process, the accuracy of the application list is improved, thereby enhancing the accuracy of determining whether a target application is an application on the user terminal.

[0074] Furthermore, after the vehicle terminal and the user terminal establish a communication connection, the vehicle terminal can send an application acquisition request to the communication interconnection application on the user terminal through the communication interconnection application on the vehicle terminal; the user terminal can also send attribute information of the application of the target type to the vehicle terminal through the communication interconnection application on the user terminal.

[0075] S402 stores the application list in the memory space corresponding to the voice application.

[0076] In this embodiment, after receiving the application list, the vehicle terminal stores the application list in the memory space corresponding to the voice application to improve the efficiency of obtaining the application list during voice interaction and thus improve the efficiency of voice interaction.

[0077] S403 receives voice commands, which instruct the target page element to perform the target operation.

[0078] S404, in response to a voice command, determines whether the target application is an application on the vehicle terminal.

[0079] S405, if the target application is not an application on the vehicle terminal, then obtain the application list corresponding to the user terminal and determine whether the application list contains the attribute information of the target application.

[0080] S406, if the application manifest contains attribute information of the target application, then the target application is determined to be an application on the user terminal; otherwise, the target application is determined not to be an application on the user terminal.

[0081] S407, if the target application is an application on the user terminal and not an application on the vehicle terminal, then look for the target page element in the projected page elements.

[0082] S408, when the projected page elements include the target page elements, control the user terminal to perform the target operation on the target page elements according to the attribute information of the target page elements.

[0083] The implementation principles and technical effects of S403 to S408 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0084] In this embodiment, after establishing a communication connection between the vehicle-mounted terminal and the user terminal, an application list is obtained from the user terminal and stored in the memory space corresponding to the voice application. Upon receiving a voice command, the system can determine, based on the application list, whether the target application of the target page element indicated by the voice command belongs to an application on the user terminal, thus improving the efficiency and accuracy of application determination. If the target application is an application on the user terminal but not on the vehicle-mounted terminal, and if the projected page element includes the target page element, the system controls the user terminal to perform the target operation on the target page element according to its attribute information. In this way, while realizing voice interaction in projection display and improving the convenience and security of human-computer interaction, the system also improves the accuracy and efficiency of voice interaction in projection display.

[0085] In some embodiments, based on any of the foregoing embodiments, a possible implementation of finding a target page element among projected page elements includes: obtaining page element information, which includes attribute information of the projected page element, including identification information of the projected page element; matching the identification information of the target page element with the identification information of the projected page element in the page element information to determine whether the projected page element includes the target page element. Thus, based on page element information, the accuracy and efficiency of determining whether a target page element is projected onto the in-vehicle terminal are improved.

[0086] Figure 5This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 4 .like Figure 5 As shown, the human-computer interaction methods in projection display include:

[0087] S501 receives voice commands, which instruct the target page element to perform the target operation.

[0088] S502, in response to a voice command, determines whether the target application to which the target page element belongs is an application on the user terminal or an application on the vehicle terminal.

[0089] The implementation principles and technical effects of S501 to S502 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0090] S503, if the target application is an application on a user terminal and not an application on an in-vehicle terminal, then obtain page element information, which includes the attribute information of the projected page element, including the identification information of the projected page element.

[0091] The identification information of the projection page element may include at least one of the following: the unique number of the projection page element, the element name of the projection page element, and the pattern of the projection page element.

[0092] In this embodiment, after the vehicle-mounted terminal projects and displays the page elements, it can obtain and store a list of applications corresponding to the user terminal from the user terminal via its communication connection with the user terminal. After determining that the target application is an application on the user terminal and not on the vehicle-mounted terminal, it retrieves the list of applications corresponding to the user terminal from the storage space. Alternatively, the vehicle-mounted terminal can obtain page element information from the user terminal after determining that the target application is an application on the user terminal and not on the vehicle-mounted terminal.

[0093] S504, match the identification information of the target page element with the identification information of the projected page element in the page element information to determine whether the projected page element includes the target page element.

[0094] In this embodiment, the identification information of the target page element is matched with the identification information of the projected page element in the page element information. If there is identification information in the page element information that matches the identification information of the target page element, it can be determined that the projected page element contains the target page element; otherwise, it can be determined that the projected page element does not contain the target page element.

[0095] S505, when the projected page elements include the target page elements, control the user terminal to perform the target operation on the target page elements according to the attribute information of the target page elements.

[0096] The implementation principle and technical effects of S505 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0097] In this embodiment, when it is determined that the target application to which the target page element indicated by the voice command belongs is an application on both the user terminal and the vehicle terminal, the vehicle terminal determines whether the projected page element contains the target page element based on the page element information containing the attribute information of the projected page element. This improves the accuracy and efficiency of determining whether the projected page element contains the target page element. If the projected page element includes the target page element, the user terminal is controlled to perform the target operation on the target page element according to the attribute information of the target page element. Thus, while realizing voice interaction in projection display and improving the convenience and security of human-computer interaction, it also improves the accuracy and efficiency of voice interaction in projection display.

[0098] In some embodiments, after determining that the target application is an application on the user terminal and not an application on the vehicle terminal, the vehicle terminal can obtain page element information from the memory space corresponding to the voice application. Specifically, after projection display, the vehicle terminal can obtain page element information from the user terminal, store the page element information in the memory space corresponding to the voice application, and then, after determining that the target application is an application on the user terminal and not an application on the vehicle terminal, obtain the page element information from the memory space corresponding to the voice application. This improves the efficiency of obtaining page element information and enhances the efficiency of human-computer interaction in projection display.

[0099] In some embodiments, when the voice application is in a wake-up state, the in-vehicle terminal can obtain updated page element information from the user terminal according to an update cycle and store the updated page element information. This fully considers the possibility of changes in the projected image on the in-vehicle terminal; for example, the navigation image projected from the user terminal to the in-vehicle terminal may change in real time while the vehicle is in motion. By updating the page element information promptly according to the update cycle while in voice wake-up mode, the accuracy of the page element information is improved, thereby enhancing the accuracy of determining whether the projected page elements contain the target page element.

[0100] Users can wake up the voice application by saying a wake-up word; or, they can preset a period during which the voice application is in a wake-up state, during which the in-vehicle terminal opens and runs the voice application to keep it in a wake-up state.

[0101] In this embodiment, when the voice application is in a wake-up state, the vehicle terminal can obtain updated page element information from the user terminal once every update cycle, store the updated page element information, and update the page element information in a timely manner.

[0102] Furthermore, the in-vehicle terminal can store updated page element information in the memory space of the voice application to improve the efficiency of obtaining page element information during voice interaction.

[0103] Furthermore, when the voice application is in a wake-up state, the vehicle terminal can obtain updated page element information from the user terminal through the communication interconnection application according to the update cycle, and store the updated page element information.

[0104] In some embodiments, the projected page elements are page elements displayed on the virtual screen of the user terminal, rather than page elements displayed on the user terminal's actual display screen. After the vehicle terminal and the user terminal establish a communication connection, the user terminal can generate a virtual screen corresponding to the vehicle terminal and project the content displayed on the virtual screen onto the vehicle terminal. This virtual screen can be understood as being dedicated to projection; applications opened through the vehicle terminal will be displayed on the virtual screen without affecting the content displayed on the user terminal's actual display screen, thus improving the user experience in screen projection.

[0105] In some embodiments, the vehicle-mounted terminal can send page element retrieval requests to the user terminal according to an update cycle, while the voice application is in a wake-up state. The page element retrieval request instructs the user terminal to retrieve the page elements displayed on its virtual screen. Upon receiving the request, the user terminal can obtain the attribute information of the page elements on its own display screen, encapsulate this attribute information to obtain encapsulated page element data, and send this data to the vehicle-mounted terminal. The vehicle-mounted terminal receives the encapsulated page element data returned by the user terminal, processes it, and obtains updated page element information. Thus, by polling and periodically retrieving the attribute information of the page elements displayed on the user terminal's virtual screen while the voice application is in a wake-up state, the accuracy of the page element information is improved.

[0106] In this embodiment, when the voice application is in a wake-up state, the vehicle terminal sends a page element retrieval request to the user terminal according to the update cycle. After receiving the page element retrieval request, the user terminal, considering the difference between the operating system of the vehicle terminal and the operating system of the user terminal, encapsulates the attribute information of the page element into a format that the operating system of the vehicle terminal supports parsing, obtains page element encapsulated data, and sends the page element encapsulated data to the vehicle terminal. The vehicle terminal receives the page element encapsulated data returned by the user terminal, performs data processing such as format parsing and data reading on the page element encapsulated data, and obtains the updated page element information.

[0107] Furthermore, the vehicle-mounted terminal can send a page element retrieval request to the user terminal through the communication interconnection application on the vehicle-mounted terminal; the user terminal can send page element encapsulated data to the vehicle-mounted terminal through the communication interconnection application on the user terminal.

[0108] In some embodiments, the attribute information of the projected page element in the page element information may include the position information and size information of the projected page element. Based on this, the position information of the projected screen where the projected page element is located can be updated according to the updated page element information. Specifically, the position information of the projected screen where the projected page element is located can be determined and updated on the in-vehicle terminal according to the position information and size information of the projected page element in the updated page element information, thereby improving the position accuracy of the projected screen.

[0109] In this embodiment, the position information of the projected page element can be its position coordinates on the user terminal's display screen, and the size information of the projected page element can be its size on the user terminal's display screen. The position coordinates of the projected image on the vehicle terminal can be calculated based on the position and size information of the projected page element. For example, the position and size information of the projected page element on the vehicle terminal can have a certain proportional relationship with the position and size information of the projected page element on the user terminal's display screen. Based on this proportional relationship, the position and size information of the projected page element on the vehicle terminal can be calculated first, and then the position coordinates of the projected image on the vehicle terminal can be calculated based on this information.

[0110] This scheme is based on voice commands to instruct users to click on target page elements, and the target page elements' attribute information includes their position information. Figure 6 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 5 .like Figure 6 As shown, the human-computer interaction methods in projection display include:

[0111] S601 receives voice commands, which instruct users to click on a target page element.

[0112] S602, in response to a voice command, searches for the target page element among the projected page elements, which are the page elements projected onto the vehicle terminal by the user terminal.

[0113] The implementation principles and technical effects of S601 to S602 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0114] S603, when the projected page elements include the target page element, the location information of the target page element is sent to the user terminal to instruct the user terminal to click on the target page element based on the location information of the target page element.

[0115] The location information of the target page element can be either the position coordinates of the target page element in the projection screen of the vehicle terminal or the position coordinates of the target page element in the display screen of the user terminal.

[0116] In this embodiment, when the projected page elements include a target page element, the vehicle-mounted terminal can obtain the attribute information of the target page element from the attribute information of the view page elements, and obtain the location information of the target page element from the attribute information of the target page element. The location information of the target page element can be sent to the user terminal via a communication interconnection application, so that the user terminal can perform a click operation on the target page element based on the location information. Thus, simulated clicking of the target page element is realized on the vehicle-mounted terminal, enabling clicking of page elements in the projected display via voice.

[0117] As an example, after receiving a user's voice command, the vehicle terminal can search for the identification information of the target page element in the identification information of the projected page elements. If the identification information of the projected page element matches the identification information of the target page element, the center point coordinates of the matching projected page element can be obtained and sent to the communication interconnection application of the vehicle terminal. The communication interconnection application then initiates a simulated click operation, that is, the communication interconnection application sends the center point coordinates to the user terminal, and the user terminal clicks the target page element based on the center point coordinates.

[0118] In this embodiment of the disclosure, when a voice command instructs a click operation on a target page element and the attribute information of the target page element includes its location information, the vehicle-mounted terminal can transmit the location information of the target page element to the user terminal to simulate a click operation on the target page element. Thus, clicking on page elements via voice interaction improves the convenience and safety of human-computer interaction when clicking on page elements in a projected display.

[0119] This scheme is based on voice commands to instruct users to input content into target page elements, and the attribute information of the target page elements includes their identification information. Figure 7 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 6 .like Figure 7 As shown, the human-computer interaction methods in projection display include:

[0120] S701 receives voice commands, which instruct users to input content into target page elements.

[0121] S702, in response to voice commands, searches for the target page element among the projected page elements, which are the page elements projected onto the vehicle terminal by the user terminal.

[0122] The implementation principles and technical effects of S701 to S702 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0123] S703 obtains the input content from voice commands.

[0124] In this embodiment, the vehicle-mounted terminal can parse the voice commands to obtain the content to be input.

[0125] S704 sends the content to be input and the identification information of the target page element to the user terminal, instructing the user terminal to input the content to be input into the target page element.

[0126] The target page element can be an input box, selection box, or other control that supports input content. The identification information of the target page element can include at least one of the following: the unique number of the target page element, the element name of the target page element, and the pattern of the target page element.

[0127] In this embodiment, the vehicle-mounted terminal can send the content to be input and the identification information of the target page element to the user terminal. After receiving the content to be input and the identification information of the target page element, the user terminal can search for the target page element among the multiple page elements displayed on its own according to the identification information of the target page element, and then input the content to be input into the target page element.

[0128] As an example, if the voice command is "Enter destination city A", then the voice command instructs the user to enter "city A" into the destination input box. The target page element is the destination input box, and the content to be entered is "city A". The vehicle terminal can send the identifier information of the destination input box and the input content "city A" to the user terminal, so that the user terminal can enter "city A" into the destination input box.

[0129] In this embodiment of the disclosure, when a voice command instructs the user to input content onto a target page element, and the target page element's attribute information includes its identifier, the vehicle-mounted terminal can transmit the content to be input and the target page element's identifier to the user terminal, allowing the user terminal to input the content onto the target page element. This voice-based content input improves the convenience and user experience of human-computer interaction when inputting content onto projected page elements, thereby enhancing driving safety.

[0130] Figure 8 This is a flowchart illustrating the human-computer interaction method in projection display provided according to embodiments of the present disclosure. Figure 7 .like Figure 8 As shown, the human-computer interaction methods in projection display include:

[0131] S801 receives voice commands, which instruct the target page element to perform the target operation.

[0132] S802 responds to voice commands by searching for the target page element among the projected page elements, which are the page elements projected onto the vehicle terminal by the user terminal.

[0133] The implementation principles and technical effects of S801 to S802 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0134] S803, if the projected page elements do not include the target page elements, an application launch command is sent to the user terminal, which instructs the user terminal to launch the target application to which the target page elements belong.

[0135] In this embodiment, the projected page elements do not include the target page element, indicating that the target page element is not projected onto the vehicle terminal and the target application to which the target page element belongs is not launched. Therefore, when the projected page elements do not include the target page element, the vehicle terminal sends an application launch command to the user terminal. The application launch command contains the identification information of the target application. After receiving the application launch command, the user terminal can obtain the identification information of the target application from the application launch command and launch the target application according to the identification information.

[0136] In this embodiment of the disclosure, when the target page element indicated by the voice command is not projected on the vehicle terminal, the vehicle terminal can control the user terminal to launch the target application to which the target page element belongs. The launch of the application in the projected display is realized through voice interaction, which improves the convenience of human-computer interaction in launching the application and improves driving safety.

[0137] Figure 9Schematic diagram of the structure of the human-computer interaction device in the projection display provided in the embodiments of this disclosure. Figure 1 .like Figure 9 As shown, the human-computer interaction device 900 in the projection display includes:

[0138] The receiving unit 901 is used to receive voice commands, which instruct the target operation to be performed on the target page element.

[0139] The lookup unit 902 is used to search for a target page element in response to a voice command. The projected page element is the page element that is projected and displayed on the vehicle terminal by the user terminal.

[0140] The control unit 903 is used to control the user terminal to perform target operations on the target page elements based on the attribute information of the target page elements when the projected page elements include target page elements.

[0141] Figure 10 Schematic diagram of the structure of the human-computer interaction device in the projection display provided in the embodiments of this disclosure. Figure 2 .like Figure 10 As shown, the human-computer interaction device 1000 in the projection display includes:

[0142] The receiving unit 1001 is used to receive voice commands, which instruct the target operation to be performed on the target page element.

[0143] The lookup unit 1002 is used to search for a target page element in response to a voice command. The projected page element is the page element that is projected and displayed on the vehicle terminal by the user terminal.

[0144] The control unit 1003 is used to control the user terminal to perform target operations on the target page elements according to the attribute information of the target page elements when the projected page elements include target page elements.

[0145] In some embodiments, the search unit 1002 includes: an application determination module 10021, configured to determine whether the target application to which the target page element belongs is an application on a user terminal or an application on an in-vehicle terminal; and a page element search module 10022, configured to search for the target page element in the projected page elements if the target application is an application on a user terminal and not an application on an in-vehicle terminal.

[0146] In some embodiments, the application determination module 10021 includes: a first determination submodule (not shown in the figure), used to determine whether the target application is an application on the vehicle terminal; a list acquisition submodule (not shown in the figure), used to acquire an application list corresponding to the user terminal if the target application is not an application on the vehicle terminal, the application list containing attribute information of multiple applications on the user terminal; a second determination submodule (not shown in the figure), used to determine whether the application list contains attribute information of the target application; and an application determination submodule (not shown in the figure), used to determine that the target application is an application on the user terminal if the application list contains attribute information of the target application, otherwise determine that the target application is not an application on the user terminal.

[0147] In some embodiments, the list acquisition submodule is specifically used to: acquire an application list from the memory space corresponding to the voice application; wherein the voice application is used to execute voice interaction functions on the vehicle terminal.

[0148] In some embodiments, the human-computer interaction device in the projection display further includes: a list acquisition unit 1004, used to acquire an application list from the user terminal after the vehicle terminal and the user terminal establish a communication connection; and a list storage unit 1005, used to store the application list in the memory space corresponding to the voice application.

[0149] In some embodiments, the list acquisition unit 1004 includes: an application acquisition request sending module (not shown in the figure), used to send an application acquisition request to a user terminal, the application acquisition request being used to request the acquisition of attribute information of an application of the target type; and a list receiving module (not shown in the figure), used to receive an application list returned by the user terminal in response to the application acquisition request, wherein the application list includes attribute information of applications of the target type on the user terminal.

[0150] In some embodiments, the search unit 1002 includes: a page element acquisition module 10023, used to acquire page element information, the page element information including attribute information of the projected page element, the attribute information of the projected page element including the identification information of the projected page element; and a page element matching module 10024, used to match the identification information of the target page element with the identification information of the projected page element in the page element information to determine whether the projected page element includes the target page element.

[0151] In some embodiments, the page element acquisition module 10023 includes: a page element acquisition submodule (not shown in the figure), used to acquire page element information from the memory space corresponding to the voice application; wherein the voice application is used to execute the voice interaction function on the vehicle terminal.

[0152] In some embodiments, the human-computer interaction device in the projection display further includes: a page element updating unit 1006, used to obtain updated page element information from the user terminal according to the update cycle when the voice application is in a wake-up state; and a page element storage unit 1007, used to store the updated page element information into memory space.

[0153] In some embodiments, the page element update unit 1006 includes: an element acquisition request sending module (not shown in the figure), configured to send a page element acquisition request to the user terminal according to an update cycle when the vehicle terminal is in voice wake-up mode, the page element acquisition request indicating the acquisition of page elements displayed on the virtual screen of the user terminal; an element receiving module (not shown in the figure), configured to receive page element encapsulation data returned by the user terminal; and a data processing module (not shown in the figure), configured to perform data processing on the page element encapsulation data to obtain updated page element information.

[0154] In some embodiments, the human-computer interaction device in the projection display further includes: a position update unit 1008, used to update the position information of the projection screen where the projection page element is located according to the updated page element information.

[0155] In some embodiments, a voice command instructs a click operation on a target page element. The attribute information of the target page element includes the location information of the target page element. The control unit 1003 includes a location sending module 10031, which is used to send the location information of the target page element to the user terminal to instruct the user terminal to click on the target page element based on the location information of the target page element.

[0156] In some embodiments, a voice command instructs a content input operation on a target page element. The attribute information of the target page element includes the identification information of the target page element. The control unit 1003 includes: an input content acquisition module 10032, used to acquire the content to be input from the voice command; and a content identification sending module 10033, used to send the content to be input and the identification information of the target page element to the user terminal, so as to instruct the user terminal to input the content to be input onto the target page element.

[0157] In some embodiments, the human-computer interaction device in the projection display further includes: an application launch unit 1009, configured to send an application launch instruction to a user terminal when the projected page elements do not include a target page element, the application launch instruction instructing the user terminal to launch the target application to which the target page element belongs.

[0158] Figures 9-10 The human-computer interaction device in the provided projection display can execute the corresponding method embodiments described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0159] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the scheme provided in any of the above embodiments.

[0160] According to embodiments of this disclosure, this disclosure also provides an autonomous driving vehicle, which includes the electronic devices provided in the foregoing embodiments. The processor in the electronic devices of the autonomous driving vehicle is capable of executing the solutions provided in any of the foregoing embodiments.

[0161] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the scheme provided in any of the above embodiments.

[0162] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0163] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0164] Figure 11This is a schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0165] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can make decisions based on data stored in a read-only memory (ROM). Figure 11 Taking ROM 1102 as an example, the computer program is loaded from storage unit 1108 into the random access memory (RAM). Figure 11 The computer program (using RAM 1103 as an example) is used to perform various appropriate actions and processes. RAM 1103 can also store various programs and data required for the operation of electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface ( Figure 11 (Taking I / O interface 1105 as an example) is also connected to bus 1104.

[0166] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0167] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as human-computer interaction methods in projection displays. For example, in some embodiments, the human-computer interaction methods in projection displays can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the human-computer interaction methods in projection displays described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform human-computer interaction methods in a projected display by any other suitable means (e.g., by means of firmware).

[0168] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0169] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0170] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0171] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0172] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0173] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0174] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0175] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A human-computer interaction method in projection display, comprising: Receive a voice command, which instructs to perform a target operation on a target page element; In response to the voice command, the target page element is searched among the projected page elements, where the projected page element is the page element that is projected and displayed on the vehicle terminal by the user terminal. When the projected page element includes the target page element, the vehicle terminal obtains the attribute information of the target page element from the attribute information of the projected page element, sends the attribute information of the target page element and the target operation to the user terminal, and controls the user terminal to perform the target operation on the target page element according to the attribute information of the target page element; The step of searching for the target page element among the projected page elements includes: Determine whether the target application to which the target page element belongs is an application on the user terminal or an application on the vehicle terminal; If the target application is an application on the user terminal and not an application on the vehicle terminal, then the target page element is searched among the projected page elements; The method further includes: When the voice application is in a wake-up state, updated page element information is obtained from the user terminal according to the update cycle; The updated page element information is stored in the memory space corresponding to the voice application so that when it is determined that the target application is an application on the user terminal and not an application on the vehicle terminal, the page element information can be obtained from the memory space corresponding to the voice application. The attribute information of the projected page element includes the position information and size information of the projected page element. Based on the position information and size information of the projected page element in the updated page element information, the position information of the projected screen where the projected page element is located is determined and updated on the vehicle terminal.

2. The human-computer interaction method in projection display according to claim 1, wherein, Determining whether the target application to which the target page element belongs is an application on the user terminal or an application on the vehicle terminal includes: Determine whether the target application is an application on the vehicle terminal; If the target application is not an application on the vehicle terminal, then obtain the application list corresponding to the user terminal, which contains attribute information of multiple applications on the user terminal; Determine whether the application list contains the attribute information of the target application; If the application list contains attribute information of the target application, then the target application is determined to be an application on the user terminal; otherwise, the target application is determined not to be an application on the user terminal.

3. The human-computer interaction method in projection display according to claim 2, wherein, Obtaining the list of applications corresponding to the user terminal includes: Obtain the application list from the memory space corresponding to the voice application; The voice application is used to perform voice interaction functions on the in-vehicle terminal.

4. The human-computer interaction method in projection display according to claim 3 further includes: After the vehicle terminal establishes a communication connection with the user terminal, the application list is obtained from the user terminal; The application list is stored in the memory space corresponding to the voice application.

5. The human-computer interaction method in projection display according to claim 4, wherein, Obtaining the application list from the user terminal includes: Send an application acquisition request to the user terminal, wherein the application acquisition request is used to request the acquisition of attribute information of an application of the target type; The user terminal receives the application list returned in response to the application acquisition request, wherein the application list includes attribute information of applications on the user terminal whose application type is the target type.

6. The human-computer interaction method in projection display according to any one of claims 1 to 5, wherein, Locating the target page element within the projected page elements includes: Obtain page element information, the page element information including the attribute information of the projected page element, the attribute information of the projected page element including the identification information of the projected page element; The identification information of the target page element is matched with the identification information of the projected page element in the page element information to determine whether the projected page element includes the target page element.

7. The human-computer interaction method in projection display according to claim 6, wherein, The process of obtaining page element information includes: The page element information is obtained from the memory space corresponding to the voice application; The voice application is used to perform voice interaction functions on the in-vehicle terminal.

8. The human-computer interaction method in projection display according to claim 7, wherein, When the voice application is in an active state, obtaining updated page element information from the user terminal according to the update cycle includes: When the voice application is in a wake-up state, a page element retrieval request is sent to the user terminal according to the update cycle. The page element retrieval request indicates the retrieval of page elements displayed on the virtual screen of the user terminal. Receive page element encapsulation data returned by the user terminal; The data encapsulated in the page elements is processed to obtain updated page element information.

9. The human-computer interaction method in projection display according to any one of claims 1 to 5, wherein, The voice command instructs a click operation on the target page element, the attribute information of the target page element includes the position information of the target page element, and controlling the user terminal to perform the target operation on the target page element according to the attribute information of the target page element includes: The location information of the target page element is sent to the user terminal to instruct the user terminal to click on the target page element based on the location information.

10. The human-computer interaction method in projection display according to any one of claims 1 to 5, wherein, The voice command instructs the user terminal to input content onto the target page element. The attribute information of the target page element includes its identifier. Controlling the user terminal to perform the target operation on the target page element based on its attribute information includes: Obtain the content to be input from the voice command; The system sends the content to be input and the identification information of the target page element to the user terminal to instruct the user terminal to input the content to be input into the target page element.

11. The human-computer interaction method in projection display according to any one of claims 1 to 5, further comprising: If the projected page element does not include the target page element, an application launch command is sent to the user terminal, which instructs the user terminal to launch the target application to which the target page element belongs.

12. A human-computer interaction device for projection display, comprising: A receiving unit is configured to receive a voice command, which instructs a target operation to be performed on a target page element. The search unit is used to search for the target page element in response to the voice command, wherein the projected page element is a page element that is projected and displayed on the vehicle terminal by the user terminal. The control unit is configured to, when the projected page element includes the target page element, obtain the attribute information of the target page element from the attribute information of the projected page element, send the attribute information of the target page element and the target operation to the user terminal, and control the user terminal to perform the target operation on the target page element according to the attribute information of the target page element; The search unit includes: The application determination module is used to determine whether the target application to which the target page element belongs is an application on the user terminal or an application on the vehicle terminal; The page element search module is used to search for the target page element in the projected page elements if the target application is an application on the user terminal and not an application on the vehicle terminal. The page element update unit is used to obtain updated page element information from the user terminal according to the update cycle when the voice application is in a wake-up state. A page element storage unit is used to store updated page element information into memory space so that when it is determined that the target application is an application on the user terminal and not an application on the vehicle terminal, the page element information can be obtained from the memory space corresponding to the voice application. The attribute information of the projection page element includes the position information and size information of the projection page element. The position update unit is used to determine and update the position information of the projection screen where the projection page element is located on the vehicle terminal based on the position information and size information of the projection page element in the updated page element information.

13. The human-computer interaction device in a projection display according to claim 12, wherein, The application judgment module includes: The first judgment submodule is used to determine whether the target application is an application on the vehicle terminal; The list acquisition submodule is used to acquire the application list corresponding to the user terminal if the target application is not an application on the vehicle terminal. The application list contains attribute information of multiple applications on the user terminal. The second judgment submodule is used to determine whether the application list contains the attribute information of the target application; The application determination submodule is configured to determine that the target application is an application on the user terminal if the application list contains attribute information of the target application, otherwise determine that the target application is not an application on the user terminal.

14. The human-computer interaction device in a projection display according to claim 13, wherein, The list acquisition submodule is specifically used for: Obtain the application list from the memory space corresponding to the voice application; The voice application is used to perform voice interaction functions on the in-vehicle terminal.

15. The human-computer interaction device in the projection display according to claim 14, further comprising: The list acquisition unit is used to acquire the application list from the user terminal after the vehicle terminal establishes a communication connection with the user terminal; The manifest storage unit is used to store the application manifest in the memory space corresponding to the voice application.

16. The human-computer interaction device in a projection display according to claim 15, wherein, The inventory acquisition unit includes: The application acquisition request sending module is used to send an application acquisition request to the user terminal. The application acquisition request is used to request the acquisition of attribute information of the application of the target type. The list receiving module is used to receive the application list returned by the user terminal in response to the application acquisition request, wherein the application list includes attribute information of applications on the user terminal whose application type is the target type.

17. The human-computer interaction device in a projection display according to any one of claims 12 to 16, wherein, The search unit includes: A page element acquisition module is used to acquire page element information, the page element information including the attribute information of the projected page element, the attribute information of the projected page element including the identification information of the projected page element; The page element matching module is used to match the identification information of the target page element with the identification information of the projected page element in the page element information to determine whether the projected page element includes the target page element.

18. The human-computer interaction device in a projection display according to claim 17, wherein, The page element acquisition module includes: The page element acquisition submodule is used to acquire the page element information from the memory space corresponding to the voice application; The voice application is used to perform voice interaction functions on the in-vehicle terminal.

19. The human-computer interaction device in a projection display according to claim 18, wherein, The page element update unit includes: The element acquisition request sending module is used to send a page element acquisition request to the user terminal according to the update cycle when the voice application is in the wake-up state. The page element acquisition request indicates the acquisition of page elements displayed on the virtual screen of the user terminal. An element receiving module is used to receive page element encapsulation data returned by the user terminal; The data processing module is used to process the encapsulated data of the page elements to obtain updated page element information.

20. The human-computer interaction device in a projection display according to any one of claims 12 to 16, wherein, The voice command instructs a click operation on the target page element, the attribute information of the target page element includes the position information of the target page element, and the control unit includes: The location sending module is used to send the location information of the target page element to the user terminal, so as to instruct the user terminal to click on the target page element based on the location information of the target page element.

21. The human-computer interaction device in a projection display according to any one of claims 12 to 16, wherein, The voice command instructs the input of content to the target page element, the attribute information of the target page element includes the identification information of the target page element, and the control unit includes: The input content acquisition module is used to acquire the content to be input from the voice command; The content identifier sending module is used to send the content to be input and the identifier information of the target page element to the user terminal, so as to instruct the user terminal to input the content to be input into the target page element.

22. The human-computer interaction device in a projection display according to any one of claims 12 to 16, further comprising: An application launch unit is configured to send an application launch instruction to the user terminal when the projected page element does not include the target page element, the application launch instruction instructing the user terminal to launch the target application to which the target page element belongs.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the human-computer interaction method in the projection display as described in any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the human-computer interaction method in the projection display as described in any one of claims 1 to 11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the human-computer interaction method in a projection display as claimed in any one of claims 1 to 11.

26. An intelligent driving vehicle, including the electronic equipment as described in claim 23.

Images