Control method, device, and storage medium for target vehicle

Abstract

The present disclosure provides a target vehicle control method, device and storage medium, relates to the technical field of data processing, in particular to the technical field of artificial intelligence, voice technology, image processing and the like. The specific implementation scheme is: in response to a trigger instruction, the interface features of an interface in a display screen are determined; wherein the display screen is a display screen displayed by a target vehicle; an element recognition mode matching the interface features of the interface is selected; based on the element recognition mode matching the interface features of the interface, element recognition is performed on the interface in the display screen, and a plurality of interface elements are obtained.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of artificial intelligence, voice technology and image processing. Background Technology

[0002] With the advancements in voice recognition technology, voice recognition is becoming increasingly common in various scenarios. For example, in driving situations, voice recognition can be used to meet user needs, thereby enhancing both the level of intelligence and driving safety. However, in this scenario, achieving more accurate recognition has become one of the factors affecting user experience. Summary of the Invention

[0003] This disclosure provides a method, device, and storage medium for controlling a target vehicle.

[0004] According to one aspect of this disclosure, a method for controlling a target vehicle is provided, comprising:

[0005] In response to a trigger command, the interface features of the interface on the display screen are determined; wherein, the display screen is the display screen shown by the target vehicle;

[0006] Select the element recognition pattern that matches the interface features of the interface.

[0007] Based on the element recognition pattern that matches the interface features, the elements of the interface on the display screen are identified to obtain multiple interface elements.

[0008] According to another aspect of this disclosure, an in-vehicle device is provided, comprising:

[0009] An instruction processing unit is configured to determine the interface features of the interface on the display screen in response to a trigger instruction; wherein the display screen is the display screen shown by the target vehicle.

[0010] An element recognition unit is used to select an element recognition pattern that matches the interface features of the interface; based on the element recognition pattern that matches the interface features of the interface, the element recognition is performed on the interface in the display screen to obtain multiple interface elements.

[0011] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0012] At least one processor; and

[0013] The memory is communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0015] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0016] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0017] Thus, the present invention can use an element recognition pattern that matches the interface features to identify elements of the interface, thereby effectively improving recognition capability and efficiency.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] Figure 1 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 1 ;

[0020] Figure 2 This is a schematic diagram of the interface display effect of the display screen of the target vehicle according to an embodiment of this application. Figure 1 ;

[0021] Figure 3 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 2 ;

[0022] Figure 4 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 3 ;

[0023] Figure 5(a) is a schematic diagram of the interface display effect of the display screen of the target vehicle according to an embodiment of the present application. Figure 2 ;

[0024] Figure 5(b) is a schematic flowchart of a control method for a target vehicle according to an embodiment of this application. Figure 4 ;

[0025] Figures 6(a) and 6(b) are schematic diagrams showing the display effect of overlapping between interfaces in the display screen of a target vehicle according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a vehicle-mounted device according to an embodiment of this application;

[0027] Figure 8 This is a block diagram of an electronic device used to implement the control method for the target vehicle in the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0030] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can still be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0031] In the automotive industry, to achieve the "see-and-talk" effect, it's necessary to identify all interface elements on the display screen. Furthermore, the comprehensiveness of interface element recognition directly impacts the user experience. Currently, the recognition rate for interface elements is relatively low.

[0032] Based on this, the present disclosure provides a control method for a target vehicle that greatly improves the recognition capability and efficiency of interface elements while achieving the "what you see is what you speak" function. Furthermore, the present disclosure requires no modifications to the target vehicle itself; it can be used both offline and online, thus effectively enhancing the user experience without significantly increasing costs.

[0033] Figure 1 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 1 This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0034] Furthermore, the method includes at least a portion of the following: For example... Figure 1 As shown, it includes:

[0035] Step S101: In response to the trigger command, determine the interface features of the interface on the display screen.

[0036] Here, the display screen refers to the display screen shown by the target vehicle, for example, such as... Figure 2 The image shows the display screen of the onboard equipment of the target vehicle.

[0037] Step S102: Select an element recognition pattern that matches the interface features of the interface.

[0038] Step S103: Based on the element recognition pattern that matches the interface features of the interface, perform element recognition on the display screen to obtain multiple interface elements.

[0039] In this way, the present solution can use an element recognition pattern that matches the interface features to identify elements of the interface, thus effectively improving recognition capability and efficiency. This lays the foundation for the subsequent realization of the "what you see is what you can say" function and provides technical support for improving user experience.

[0040] In a specific example, the interface features may include interface attributes, which may indicate whether the interface is a native interface or a web page interface, etc.; furthermore, the interface features may also include interface information, etc., and this disclosure does not impose specific limitations on this.

[0041] It should be noted that, in a specific example, the element recognition mode may be a recognition model; or, the element recognition mode may be a recognition tool for recognizing interface elements; or, the element recognition mode may be a specific recognition processing logic (e.g., specific code), which is not limited in this disclosure.

[0042] Furthermore, in a specific example, the interface element may be an element with interactive functionality, such as the application icon or control in the main interface, or the interactive function button or control in the application or webpage.

[0043] Figure 3 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 2 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figure 1 The methods shown can also be applied to this example, and the related content will not be elaborated further in this example.

[0044] Furthermore, the method includes at least a portion of the following: For example... Figure 3 As shown, it includes:

[0045] Step S301: In response to the target audio, generate a trigger command.

[0046] In a specific example, a trigger command will only be generated if the target audio meets the intent conditions. For example, if the target audio is a "what you see can be said" intent, then a trigger command will be generated to respond to that "what you see can be said" intent.

[0047] Here, the target audio is used to indicate control operations on the target element in the display screen.

[0048] Step S302: In response to the trigger command, determine the interface features of the interface on the display screen.

[0049] Here, the display screen refers to the display screen shown by the target vehicle.

[0050] Step S303: Select an element recognition pattern that matches the interface features of the interface.

[0051] Step S304: Based on the element recognition pattern that matches the interface features of the interface, perform element recognition on the display screen to obtain multiple interface elements.

[0052] Step S305: Determine the target element corresponding to the target audio from the identified plurality of interface elements.

[0053] Step S306: Based on the position information of the target element (e.g., the position information of the target element in the display screen), perform control operations on the target element.

[0054] It should be noted that during the process of identifying interface elements, the position information of each interface element can also be identified simultaneously. For example, the position information of each interface element on the display screen, or the position information of each interface element on its own interface can be identified first, and then the position information of each interface element on the display screen can be obtained. In this way, the foundation is laid for the subsequent rapid response to the target audio and the realization of the function of "what you see can be spoken".

[0055] This example provides a specific scenario supporting "what you see can be spoken". For instance, when a target audio for the "what you see can be spoken" function is detected, the present invention can be applied to identify interface elements in the display screen and determine the target element to be controlled from the identified interface elements to complete the "what you see can be spoken" function. In other words, in this example, after the target audio is detected, firstly, a trigger command is generated to determine the interface features of the display screen; secondly, an element recognition mode matching the interface features is selected, and the interface in the display screen is identified using the element recognition mode to obtain multiple interface elements; finally, the target element corresponding to the target audio is determined from the multiple interface elements (for example, the position information of the target element can also be determined) to control the target element.

[0056] Thus, this disclosed solution provides a specific method for realizing the "what you see is what you can say" function. In this way, it can quickly respond to the user's target audio and perform control operations on the target elements in the target vehicle's display screen, thereby improving the intelligence level of the target vehicle and enhancing the user experience.

[0057] Furthermore, since the present invention can identify elements of the interface by using an element recognition pattern that matches the interface features, more and more comprehensive interface elements are identified, thus further improving the user experience.

[0058] Furthermore, in a specific example, after detecting the target audio, the communication conditions corresponding to the target vehicle can also be detected, thus further refining the use case and providing technical support for further improving the user experience. Specifically, the above-described response to the target audio to generate a trigger command (such as step S301 mentioned above) can specifically include:

[0059] Step S301-1: In response to the target audio, detect the communication conditions corresponding to the target vehicle;

[0060] Step S301-2: If it is determined that the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is generated.

[0061] Here, the preset requirement is that the network signal corresponding to the target vehicle is in a normal state, or that the network signal corresponding to the target vehicle is in a normal state and the signal strength is greater than the signal threshold. It is understood that the preset requirement can be adjusted based on the actual application scenario, and this disclosure does not impose specific restrictions on it.

[0062] In other words, after detecting the target audio, it is also necessary to detect the communication conditions corresponding to the target vehicle. Only when the communication conditions corresponding to the target vehicle do not meet the preset requirements, such as when the network signal is abnormal, will a trigger command be generated. At this time, in the offline state, the target vehicle's own capabilities are used to identify the interface elements in the display screen.

[0063] Thus, this disclosed solution provides a detailed scenario that supports "what you see can be spoken". For example, after detecting the target audio for which the "what you see can be spoken" function is to be implemented, it is also necessary to detect the communication conditions corresponding to the target vehicle. If the communication conditions do not meet the preset requirements, the recognition interface elements are triggered. In this way, the "what you see can be spoken" function can be implemented using the target vehicle's own capabilities in an offline state, which is more practical and further improves the intelligence level of the target vehicle. At the same time, it also further improves the user experience.

[0064] Furthermore, in another specific example, if it is determined that the communication conditions corresponding to the target vehicle meet the preset requirements, the target audio can be sent, for example, to a cloud server, and then the cloud server's recognition capability can be used to quickly respond to the target audio, so as to realize the function of "what you see can be spoken".

[0065] Figure 4 This is an illustrative flow diagram of a control method for a target vehicle according to an embodiment of this application. Figure 3 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figure 1 The methods shown can also be applied to this example, and the related content will not be elaborated further in this example.

[0066] Furthermore, the method includes at least a portion of the following: For example... Figure 4 As shown, it includes:

[0067] Step S401: If the communication conditions corresponding to the target vehicle do not meet the preset requirements, generate a trigger command.

[0068] Here, the preset requirement is that the network signal corresponding to the target vehicle is in a normal state, or that the network signal corresponding to the target vehicle is in a normal state and the signal strength is greater than the signal threshold. It is understood that the preset requirement can be adjusted based on the actual application scenario, and this disclosure does not impose specific restrictions on it.

[0069] In other words, if the communication conditions corresponding to the target vehicle are not met, such as when the network signal is detected to be abnormal, a trigger command is generated directly. At this time, regardless of whether the target audio for the "what you see can be spoken" function is detected, a trigger command needs to be generated. In this way, in the offline state, the target vehicle's own capabilities are used to identify the interface elements in the display screen in advance, laying the foundation for quickly responding to the target audio and realizing the "what you see can be spoken" function.

[0070] Step S402: In response to the trigger command, determine the interface features of the interface on the display screen.

[0071] Here, the display screen refers to the display screen shown by the target vehicle.

[0072] Step S403: Select an element recognition pattern that matches the interface features of the interface.

[0073] Step S404: Based on the element recognition pattern that matches the interface features of the interface, perform element recognition on the display screen to obtain multiple interface elements.

[0074] This example provides another specific scenario that supports what you see is what you can say. For example, if the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is directly generated to identify the interface elements in the display screen and obtain multiple interface elements.

[0075] Thus, this disclosed solution provides a specific method for obtaining multiple interface elements when the communication conditions do not meet the preset requirements. This lays the foundation for the subsequent implementation of the "what you see is what you can say" function and provides technical support for improving the user experience.

[0076] Furthermore, since the present disclosure can identify elements of the interface by using an element recognition pattern that matches the interface features, the identified interface elements are more numerous and comprehensive, thus further supporting the improvement of user experience.

[0077] Furthermore, in a specific example, after obtaining multiple interface elements when the communication conditions corresponding to the target vehicle do not meet preset requirements, for example, after step S404 described above, the method further includes:

[0078] Step S405: In response to the target audio, determine the target element corresponding to the target audio from the identified plurality of interface elements.

[0079] Here, the target audio is used to indicate control operations on the target element in the display screen.

[0080] In a specific example, a trigger command will only be generated if the target audio meets the intent conditions. For example, if the target audio is a "what you see can be said" intent, then a trigger command will be generated to respond to that "what you see can be said" intent.

[0081] Step S406: Based on the position information of the target element (e.g., the position information of the target element in the display screen), perform control operations on the target element.

[0082] It should be noted that during the process of identifying interface elements, the position information of each interface element can also be identified simultaneously. For example, the position information of each interface element on the display screen, or the position information of each interface element on its own interface can be identified first, and then the position information of each interface element on the display screen can be obtained. In this way, the foundation is laid for the subsequent rapid response to the target audio and the realization of the function of "what you see can be spoken".

[0083] Thus, this disclosure provides a specific solution for realizing the "what you see is what you speak" function. This allows for rapid response to the user's target audio and execution of control operations on target elements on the target vehicle's display screen, thereby enhancing the vehicle's intelligence and improving the user experience. Furthermore, because this disclosure utilizes an element recognition pattern that matches the interface features, it identifies more and more comprehensive interface elements, further enhancing the user experience.

[0084] In a specific example of the disclosed solution, multiple interface elements can be identified using the following method. Specifically, the element identification pattern based on the interface features of the interface described above is used to identify elements in the display screen to obtain multiple interface elements (such as step S103, step S304, or step S404 described above), specifically including at least one of the following:

[0085] Method 1: If it is determined that the interface belongs to the native interface based on the interface features of the interface, at least based on the first identification mode, the identification information of the multiple interface elements contained in the interface is obtained; wherein, the first identification mode is used to call the interface of the display system of the display screen to obtain the identification information of the interface elements contained in the interface that belong to the native interface.

[0086] In other words, when the interface is a native interface, a first recognition mode that matches the native interface is used to recognize the interface in order to obtain multiple interface elements.

[0087] It should be noted that the native interface can be the interface built into the system corresponding to the display screen, such as the system interface in the Android system. In this scenario, pre-set function services of the system corresponding to the display screen can be invoked, such as accessibility services, and the identification information of all interface elements in the native interface can be obtained using system interfaces.

[0088] It should be noted that the system's pre-set functional services, such as accessibility services, can obtain the identification information of the controls built into the system, such as the identification information of the system's built-in applications.

[0089] Method 2: If the interface is determined to be a program interface of a preset webpage or preset application based on the interface features of the interface, at least based on the second recognition mode, the interface elements with interactive functions (such as controls, function buttons, etc. with interactive functions) in the interface are determined; wherein, the second recognition mode is used to obtain the configuration file corresponding to the interface (for example, obtain the configuration file of the interface that has been pre-configured), and the configuration file contains the interface elements with interactive functions contained in the interface.

[0090] In other words, when the interface displayed on the screen is a preset webpage or a preset application's program interface, a second identification mode can be selected to obtain the configuration file corresponding to the preset webpage or preset application's program interface. For example, when the interface is a preset webpage, the configuration file of the preset webpage can be obtained; or when the interface is a preset application's program interface, the configuration file of the preset application's program interface can be obtained, thereby facilitating the extraction of the interactive interface elements contained in the interface from the obtained configuration file.

[0091] It should be noted that for interactive interface elements in the program interface of a preset webpage or preset application, some interface elements, such as custom ImageView toggle buttons, may not be recognized by the accessibility service built into the Android system. In this case, a corresponding configuration file can be configured in advance for the preset webpage and / or the program interface of the preset application, and then the interactive interface elements contained in the program interface of the preset webpage or the preset application can be obtained using the pre-configured configuration file.

[0092] In a specific example, the configuration file can be obtained by following these steps:

[0093] Step S501: Use the interface element identification (ID) tool to obtain the identification information of each interface element in the specified interface (such as a preset webpage or the program interface of a preset application).

[0094] Step S502: Obtain the interface description information of the specified interface, wherein the description information includes the interface elements contained in the specified interface; for example, use the adb command to obtain the interface description information of the specified interface.

[0095] Step S503: Configure the interface description information of the specified interface obtained in step S502, and the identification information corresponding to the interface elements obtained in step 1, in the target file, such as a JSON file. Furthermore, element description information for each interface element can also be configured.

[0096] In other words, the target file may contain the interface description information of the specified interface, the identification information of each interface element identified in step S501, and the element description information of each interface element.

[0097] Step S504: Scan and obtain the label information of each interface element in the specified interface, the element description information of each interface element, etc.

[0098] Step S505: Match the identification information of each interface element obtained in step S504 with the identification information of each interface element contained in the target file obtained in step S503.

[0099] Step S506: If the identification information is successfully matched, perform control operations on the successfully matched interface elements to verify whether the control operations are successful.

[0100] Step S507: If the identification information matching fails, the element description information of each interface element obtained in step S504 is matched with the element description information of each interface element contained in the target file obtained in step S503. If the element description information matching is successful, the control operation is performed on the successfully matched interface element to verify whether the control operation is successful.

[0101] Step S508: Based on the verification results of steps S506 and S507, adjust the target file to obtain the configuration file for the specified interface. This ensures that the configuration file contains all the interface elements included in the specified interface, laying the foundation for the subsequent implementation of the "what you see is what you can say" function, and providing technical support for improving the user experience.

[0102] It should be noted that when the interface belongs to a preset webpage or preset application program interface, the first recognition mode can also be used for element recognition. That is, in this scenario, the first recognition mode and the second recognition mode are used to recognize the elements of the interface. This makes it easier to identify more and more comprehensive interface elements, which provides support for further improving the user experience.

[0103] Thus, since the present disclosure can use an element recognition pattern that matches the interface features to identify elements of the interface, more and more comprehensive interface elements are identified, thereby further supporting the improvement of user experience.

[0104] Furthermore, in a specific example of the disclosed solution, the above-described method of determining that the interface belongs to a preset webpage or preset application program interface based on the interface features of the interface, at least based on the second recognition mode, to identify the interface elements with interactive functions in the interface (such as the second method described above), may further include:

[0105] When it is determined that the interface belongs to the program interface of a preset webpage or preset application based on the interface features of the interface, the interactive interface elements in the interface are identified based on the second recognition mode, and the interactive preset icons presented by the interface are identified based on the third recognition mode, wherein the plurality of interface elements include preset icons.

[0106] In other words, in this example, when it is determined that the interface belongs to the program interface of a preset webpage or preset application, two recognition modes, such as the second recognition mode and the third recognition mode, can be used to identify the elements of the interface that belongs to the program interface of the preset webpage or preset application. This makes it easier to identify more and more comprehensive interface elements, thus providing support for further improving the user experience.

[0107] It should be noted that the default webpage (such as a page belonging to webview / safeView) or the program interface of the default application can be understood as a non-native interface. In this case, using the second recognition mode and / or the third recognition mode to identify interface elements can identify more and more comprehensive page elements.

[0108] Furthermore, it should be noted that the preset icons with interactive functions are specifically non-native interfaces, such as icons in web pages; furthermore, the preset icons with interactive functions are image-type icons, or icons that combine text information and images.

[0109] In a specific example, the third recognition mode can be OCR (Optical Character Recognition) mode, and the specific process is as follows:

[0110] To obtain an image of the interface displayed on the screen, for example, by taking a screenshot.

[0111] Based on a preset recognition model, the interface image is recognized to obtain all interface elements contained in the interface.

[0112] It should be noted that the interface image recognized by the preset recognition model can be specifically the interface image of the interface currently displayed on the screen. In this case, the interface image may not contain all the content of the interface, for example, some content may be covered by other interfaces; or it may be a pre-stored complete interface image of the interface.

[0113] In this way, the disclosed solution can identify interface elements by using different element recognition modes for different interface features. This effectively improves the recognition capability and efficiency of interface elements, and the identified interface elements are more numerous and comprehensive. This lays the foundation for the subsequent realization of the "what you see is what you can say" function, and also provides technical support for improving user experience.

[0114] In a specific example of the disclosed solution, when there are multiple (two or more) interfaces in the display area, the interface elements of each interface can also be obtained, thereby identifying more and more comprehensive interface elements, which further provides technical support for realizing the function of "what you see is what you can say".

[0115] Specifically, the interface mentioned above is one of at least two interfaces that are in the open state displayed on the display screen; for example, as shown in Figure 5(a), the target vehicle's display screen displays three interfaces that are in the open state, namely interface 1, interface 2 and interface 3. In this case, the interface mentioned above is any one of the three interfaces.

[0116] Furthermore, in a specific example, more and more comprehensive interface elements can be obtained in the following manner; specifically, the element recognition pattern based on the interface features of the interface described above is used to identify elements of the interface on the display screen to obtain multiple interface elements (such as the steps S103, S304, or S404 described above), specifically including:

[0117] Based on the element recognition pattern that matches the interface features of each of the at least two interfaces, the interfaces in the display screen that are in the open state are identified to obtain multiple interface elements of all interfaces in the display screen that are in the open state.

[0118] In other words, for any interface that is open, an element recognition mode that matches the interface characteristics can be used to identify the interface, thereby obtaining the multiple interface elements contained in each interface.

[0119] Taking Figure 5(a) as an example, element recognition is performed on interface 1 based on the element recognition mode that matches the interface features of interface 1, element recognition is performed on interface 2 based on the element recognition mode that matches the interface features of interface 2, and element recognition is performed on interface 3 based on the element recognition mode that matches the interface features of interface 3. In this way, the interface elements contained in all interfaces in the open state on the display screen are obtained, which effectively ensures that more and more comprehensive interface elements are identified, thus laying the foundation for the subsequent realization of the "what you see is what you can say" function, and also providing technical support for improving user experience.

[0120] In a specific example, as shown in Figure 5(b), the interfaces displayed on the screen can also be identified using the three recognition modes described above simultaneously. Alternatively, if it is determined that the interface belongs to the program interface of a preset webpage or a preset application, the three recognition modes described above can be used for identification, thereby maximizing the identification of all interface elements.

[0121] In this way, the disclosed solution can identify interface elements by using different element recognition modes for different interface features. This effectively improves the recognition capability and efficiency of interface elements, and the identified interface elements are more numerous and comprehensive, laying the foundation for the subsequent realization of the "what you see is what you can say" function, thereby improving the user experience.

[0122] In a specific example of the present disclosure, the display area occupied by the interface is at least a part of the display area of ​​the display screen; for example, as shown in Figure 5(a), the display areas occupied by interface 1, interface 2 and interface 3 are all part of the entire display area of ​​the display screen.

[0123] Furthermore, the display area occupied by the interface is not covered by other interfaces among the plurality of interfaces. For example, as shown in Figure 5(a), the display areas occupied by interface 1, interface 2, and interface 3 are independent and there is no covering phenomenon. In this scenario, the present invention can identify the interface elements of each interface that is in the open state.

[0124] It should be noted that, for the scenario shown in Figure 5(a), the interface elements identified by the second recognition mode can be the interface elements displayed on the display screen, or they can be the interface elements contained in the interface but not displayed on the display screen. This disclosure does not limit this.

[0125] Alternatively, at least a portion of the display area occupied by the interface is covered by other interfaces (e.g., interfaces in the active layer). For example, as shown in Figure 6(a), the interfaces in the display screen that are active are interface 4 and interface 5, and a portion of the display area occupied by interface 4 is covered by interface 5. Here, interface 5 is the active layer interface, or can be understood as the top-level interface. Therefore, the display area occupied by interface 5 is at the top. At this time, the portion of the display area of ​​interface 4 that overlaps with interface 5 is covered by interface 5, causing a portion of interface 4 to be invisible to the user. Alternatively, as shown in Figure 6(b), the interfaces in the display screen that are active are interface 6 and interface 7, and the entire display area occupied by interface 7 is covered by interface 6. Here, interface 6 is the active layer interface, or can be understood as the top-level interface. Therefore, the display area occupied by interface 6 is at the top. At this time, the portion of the display area of ​​interface 7 that overlaps with interface 5 is covered by interface 6, causing the entire area of ​​interface 7 to be invisible to the user. Here, in the two scenarios mentioned above, in the scenario where some display areas are covered, the present solution is also able to identify the interface elements of each interface (such as interface 4 and interface 5 in Figure 6(a), or interface 6 and interface 7 in Figure 6(b)) that are in the open state.

[0126] It should be noted that, for the scenarios shown in Figure 6(a) or Figure 6(b), in the present disclosure, the interface elements in the covered display area can also be identified. For example, in Figure 6(a), the interface elements in the covered area of ​​interface 4 can also be identified. Similarly, in Figure 6(b), the interface elements in the covered area of ​​interface 7 can also be identified.

[0127] It should be noted that, for the scenarios shown in Figure 6(a) or Figure 6(b), the interface elements identified by the second recognition mode can be the interface elements displayed on the display screen, or they can be the interface elements contained in the interface but not displayed on the display screen. This disclosure does not impose any restrictions on this.

[0128] In this way, the present solution further refines the display scenarios of the interface, thereby enriching the application scenarios of the present solution, improving the recognition capability of interface elements, and improving the recognition efficiency. Moreover, the interface elements recognized are more numerous and comprehensive, laying the foundation for the subsequent realization of the "what you see is what you can say" function, and thus improving the user experience.

[0129] In a specific example of this disclosed solution, in order to further improve the user experience, the properties of the target element indicated by the target audio can also be modified. In this way, even when the target element is covered by other interfaces, the "what you see is what you can say" function can still be realized, further enriching the use cases of this disclosed solution.

[0130] Specifically, before performing control operations on the target element based on its location information, the method further includes:

[0131] Under preset conditions, based on the attribute characteristics of the target area in the first interface of the active layer, the attribute characteristics of the target element are modified to obtain the target attribute of the target element in the first interface; here, the first interface is one of at least two interfaces in the display screen that are in the open state; the target area is the area in the first interface corresponding to the target element.

[0132] Furthermore, the control operation on the target element based on the position information of the target element (such as step S306 or step S406 mentioned above) described above may specifically include:

[0133] When the attribute characteristics of the target element are the target attributes located in the first interface of the activation layer, the target element is controlled based on its position information.

[0134] In this way, the present solution modifies the attributes of the target element indicated by the target audio based on the attribute characteristics of the target area in the first interface of the activation layer. This further enriches the user's usage scenarios, lays the foundation for the subsequent implementation of the "what you see can be said" function, and further enhances the user experience.

[0135] Furthermore, in a specific example, the preset conditions include: the interface where the target element is located is an interface in an inactive layer; and the location of the target element is at least covered by a first interface in the active layer.

[0136] For example, as shown in Figure 6(b), interface 6 is the active layer interface, and the target element is located in interface 7, which is the inactive layer interface. The position of the target element is covered by the active layer interface 6. At this time, based on the attribute characteristics of the target area in the active layer interface 6, the attribute characteristics of the target element in the inactive layer interface 6 can be modified so that the position of the target element is in the active layer. This lays the foundation for subsequent control operations on the target element.

[0137] Here, the control operation can be to process the target control accordingly, such as to start, close, click or switch the target control on the display screen of the target vehicle. This disclosure does not limit the specific content of the control operation.

[0138] In this way, the disclosed solution further refines the scenarios for the "what you see is what you can say" function, further enriches the user's usage scenarios, and thus further enhances the user experience.

[0139] This disclosure also provides an in-vehicle device, such as Figure 7 As shown, it includes:

[0140] The instruction processing unit 701 is configured to determine the interface features of the interface on the display screen in response to a trigger instruction; wherein the display screen is the display screen shown by the target vehicle.

[0141] The element recognition unit 702 is used to select an element recognition pattern that matches the interface features of the interface; based on the element recognition pattern that matches the interface features of the interface, the element recognition is performed on the interface in the display screen to obtain multiple interface elements.

[0142] In a specific example of the disclosed solution, an operation unit is also included; wherein,

[0143] The instruction processing unit 701 is further configured to generate a trigger instruction in response to a target audio before determining the interface features of the interface on the display screen in response to the trigger instruction; the target audio is used to indicate control operations on the target element on the display screen.

[0144] The element recognition unit 702 is further configured to, after obtaining multiple interface elements, determine the target element corresponding to the target audio from the multiple interface elements that have been recognized;

[0145] The operation unit is used to perform control operations on the target element based on the position information of the target element.

[0146] In a specific example of the disclosed solution, the instruction processing unit 701 is specifically used for:

[0147] In response to the target audio, detect the communication conditions corresponding to the target vehicle;

[0148] If the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is generated.

[0149] In a specific example of the scheme disclosed herein, the instruction processing unit 701 is further configured to generate a trigger instruction when it is detected that the communication conditions corresponding to the target vehicle do not meet the preset requirements.

[0150] In a specific example of the disclosed solution, an operation unit is also included, wherein,

[0151] The element recognition unit is further configured to, in response to the target audio, determine the target element corresponding to the target audio from the identified plurality of interface elements; wherein the target audio is used to instruct control operations on the target element in the display screen;

[0152] The operation unit is used to perform control operations on the target element based on the position information of the target element.

[0153] In a specific example of the scheme disclosed herein, the element identification unit 702 is specifically configured to perform at least one of the following:

[0154] When it is determined that the interface belongs to the native interface based on the interface features of the interface, at least based on the first identification mode, the identification information of the interface elements contained in the interface is obtained from the multiple interface elements contained in the interface; wherein, the first identification mode is used to call the interface of the display system of the display screen to obtain the identification information of the interface elements contained in the interface that belong to the native interface.

[0155] or,

[0156] If, based on the interface features, it is determined that the interface belongs to a preset webpage or a preset application program interface, then at least based on a second identification mode, the interactive interface elements in the interface are identified; wherein, the second identification mode is used to obtain the configuration file corresponding to the interface, and the configuration file contains the interactive interface elements contained in the interface.

[0157] In a specific example of the scheme disclosed herein, the element identification unit 702 is specifically used for:

[0158] When it is determined that the interface belongs to the program interface of a preset webpage or preset application based on the interface features of the interface, the interactive interface elements in the interface are identified based on the second recognition mode, and the interactive preset icons presented by the interface are identified based on the third recognition mode, wherein the plurality of interface elements include preset icons.

[0159] In a specific example of the disclosed solution, the interface is one of at least two interfaces displayed on the display screen that are in an open state;

[0160] Specifically, the element recognition unit is used to identify the interfaces in the display screen that are in the open state based on an element recognition mode that matches the interface features of each of the at least two interfaces, thereby obtaining multiple interface elements of all interfaces in the display screen that are in the open state.

[0161] In a specific example of the scheme disclosed herein, the display area occupied by the interface is at least a portion of the display area of ​​the display screen;

[0162] The display area occupied by the interface is not covered by other interfaces among the plurality of interfaces; or, at least a portion of the display area occupied by the interface is covered by other interfaces among the plurality of interfaces.

[0163] In a specific example of the scheme disclosed herein, the operation unit is further configured to:

[0164] Under the condition of meeting the preset conditions, the attribute characteristics of the target element are modified based on the attribute characteristics of the target area in the first interface of the active layer to obtain the target attribute of the target element in the first interface; the target area is the area in the first interface corresponding to the target element; the first interface is one of at least two interfaces that are in the open state;

[0165] When the attribute characteristics of the target element are the target attributes located in the first interface of the activation layer, the target element is controlled based on its position information.

[0166] In a specific example of the disclosed solution, the preset conditions include: the interface where the target element is located is an interface in an inactive layer; and the location of the target element is at least covered by a first interface in the active layer.

[0167] For a description of the specific functions and examples of each unit of the apparatus in this disclosure embodiment, please refer to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be repeated here.

[0168] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0172] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a method for controlling a target vehicle. For example, in some embodiments, the method for controlling a target vehicle may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method for controlling a target vehicle described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the method for controlling a target vehicle by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for controlling a target vehicle, comprising: In response to a trigger command, the interface features of the interface displayed on the target vehicle's screen are determined; wherein, the interface features include interface attributes; the interface is one of at least two interfaces displayed on the screen that are in an open state; the display area occupied by the interface is not covered by other interfaces; or, at least a portion of the display area occupied by the interface is covered by other interfaces. Select the element recognition pattern that matches the interface attributes included in the interface features of the interface. Based on the element recognition pattern that matches the interface attributes included in the interface features of the interface, the interface in the display screen is identified to obtain multiple interface elements. The element recognition pattern, based on the interface attributes included in the interface features, is used to identify elements in the display screen to obtain multiple interface elements, including at least one of the following: When it is determined that the interface belongs to the native interface based on the interface attributes included in the interface features of the interface, at least based on the first identification mode, the identification information of the interface elements included in the interface is obtained from the multiple interface elements contained in the interface; wherein, the first identification mode is used to call the interface of the display system of the display screen to obtain the identification information of the interface elements included in the interface that belong to the native interface. If the interface is determined to belong to a preset webpage or preset application program interface based on the interface attributes included in the interface features of the interface, the interactive interface elements in the interface are identified at least based on the second identification mode; wherein, the second identification mode is used to obtain the configuration file corresponding to the interface, and the configuration file contains the interactive interface elements contained in the interface.

2. The method according to claim 1, wherein, Before determining the interface features of the interface on the display screen in response to a trigger command, the method further includes: In response to a target audio, a trigger command is generated; the target audio is used to instruct a control operation on a target element in the display screen. After obtaining multiple interface elements, the method further includes: The target element corresponding to the target audio is determined from the identified multiple interface elements; Based on the location information of the target element, control operations are performed on the target element.

3. The method according to claim 2, wherein, The step of generating a trigger command in response to the target audio includes: In response to the target audio, detect the communication conditions corresponding to the target vehicle; If the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is generated.

4. The method according to claim 1, further comprising: If the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is generated.

5. The method according to claim 4, further comprising: In response to a target audio, a target element corresponding to the target audio is determined from the identified plurality of interface elements; wherein the target audio is used to indicate a control operation on the target element in the display screen; Based on the location information of the target element, control operations are performed on the target element.

6. The method according to any one of claims 1-5, wherein, When the interface is determined to belong to a preset webpage or preset application program interface based on the interface attributes included in the interface features, the interactive interface elements in the interface are identified, at least based on the second recognition mode, including: When the interface is determined to belong to a preset webpage or preset application program interface based on the interface attributes included in the interface features of the interface, the interactive interface elements in the interface are identified based on the second recognition mode, and the interactive preset icons presented by the interface are identified based on the third recognition mode, wherein the plurality of interface elements include preset icons.

7. The method according to claim 2, 3 or 5, wherein, The element recognition pattern, based on the interface attributes included in the interface features, is used to identify elements in the display screen, resulting in multiple interface elements, including: Based on an element recognition pattern that matches the interface attributes included in the interface features of each of the at least two interfaces, the interfaces in the display screen that are in the open state are identified to obtain multiple interface elements of all interfaces in the display screen that are in the open state.

8. The method according to claim 7, wherein, The display area occupied by the interface is at least a portion of the display area of ​​the display screen.

9. The method according to claim 7, further comprising: Under the condition of meeting the preset conditions, the attribute characteristics of the target element are modified based on the attribute characteristics of the target area in the first interface of the active layer to obtain the target attribute of the target element in the first interface; the target area is the area in the first interface corresponding to the target element; the first interface is one of at least two interfaces that are in the open state; The step of controlling the target element based on its position information includes: When the attribute characteristics of the target element are the target attributes located in the first interface of the activation layer, the target element is controlled based on its position information.

10. The method according to claim 9, wherein, The preset conditions include: The interface where the target element is located is an inactive layer interface; the location of the target element is at least covered by the first interface in the active layer.

11. A vehicle-mounted device, comprising: An instruction processing unit is configured to, in response to a trigger instruction, determine the interface features of an interface displayed on a screen of a target vehicle; wherein the interface features include interface attributes; the interface is one of at least two interfaces displayed on the screen that are in an open state; the display area occupied by the interface is not covered by other interfaces; or, at least a portion of the display area occupied by the interface is covered by other interfaces. An element recognition unit is used to select an element recognition pattern that matches the interface attributes included in the interface features of the interface; based on the element recognition pattern that matches the interface attributes included in the interface features of the interface, the unit performs element recognition on the interface in the display screen to obtain multiple interface elements. The element identification unit is specifically used to perform at least one of the following: When it is determined that the interface belongs to the native interface based on the interface attributes included in the interface features of the interface, at least based on the first identification mode, the identification information of the interface elements included in the interface is obtained from the multiple interface elements contained in the interface; wherein, the first identification mode is used to call the interface of the display system of the display screen to obtain the identification information of the interface elements included in the interface that belong to the native interface. If the interface is determined to belong to a preset webpage or preset application program interface based on the interface attributes included in the interface features of the interface, the interactive interface elements in the interface are identified at least based on the second identification mode; wherein, the second identification mode is used to obtain the configuration file corresponding to the interface, and the configuration file contains the interactive interface elements contained in the interface.

12. The vehicle-mounted device according to claim 11, further comprising an operating unit; wherein, The instruction processing unit is further configured to generate a trigger instruction in response to a target audio before determining the interface features of the interface on the display screen in response to the trigger instruction; the target audio is used to indicate control operations on the target element on the display screen. The element recognition unit is further configured to, after obtaining multiple interface elements, determine the target element corresponding to the target audio from the identified multiple interface elements; The operation unit is used to perform control operations on the target element based on the position information of the target element.

13. The vehicle-mounted device according to claim 12, wherein, The instruction processing unit is specifically used for: In response to the target audio, detect the communication conditions corresponding to the target vehicle; If the communication conditions corresponding to the target vehicle do not meet the preset requirements, a trigger command is generated.

14. The vehicle-mounted device according to claim 11, wherein, The instruction processing unit is also used to generate a trigger instruction when it is detected that the communication conditions corresponding to the target vehicle do not meet the preset requirements.

15. The vehicle-mounted device according to claim 14, further comprising an operating unit, wherein, The element recognition unit is further configured to, in response to the target audio, determine the target element corresponding to the target audio from the identified plurality of interface elements; wherein the target audio is used to instruct control operations on the target element in the display screen; The operation unit is used to perform control operations on the target element based on the position information of the target element.

16. The vehicle-mounted device according to any one of claims 11-15, wherein, The element identification unit is specifically used for: When the interface is determined to belong to a preset webpage or preset application program interface based on the interface attributes included in the interface features of the interface, the interactive interface elements in the interface are identified based on the second recognition mode, and the interactive preset icons presented by the interface are identified based on the third recognition mode, wherein the plurality of interface elements include preset icons.

17. The vehicle-mounted device according to claim 12, 13 or 15, wherein, The element recognition unit is specifically used to identify the interfaces in the display screen that are in the open state based on an element recognition pattern that matches the interface attributes included in the interface features of each of the at least two interfaces, thereby obtaining multiple interface elements of all interfaces in the display screen that are in the open state.

18. The vehicle-mounted device according to claim 17, wherein, The display area occupied by the interface is at least a portion of the display area of ​​the display screen.

19. The vehicle-mounted device according to claim 17, wherein, The operation unit is also used for: Under the condition of meeting the preset conditions, the attribute characteristics of the target element are modified based on the attribute characteristics of the target area in the first interface of the active layer to obtain the target attribute of the target element in the first interface; the target area is the area in the first interface corresponding to the target element; the first interface is one of at least two interfaces that are in the open state; When the attribute characteristics of the target element are the target attributes located in the first interface of the activation layer, the target element is controlled based on its position information.

20. The vehicle-mounted device according to claim 19, wherein, The preset conditions include: the interface where the target element is located is an inactive layer interface; and the location of the target element is at least covered by the first interface in the active layer.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images