Interaction method based on vehicle-mounted screen

Abstract

The invention provides an interaction method and device based on a vehicle-mounted screen, electronic equipment and a storage medium, according to the embodiment of the invention, a first user gesture is recognized through a high-definition camera arranged in a cabin, and under the condition that the first user gesture correspondingly activates a gesture control mode, the gesture control mode is started; in the gesture control mode, mapping the spatial position of the hand of the user in the vehicle to a cursor position corresponding to a display screen in a vehicle cabin; and executing a corresponding vehicle control instruction according to the current cursor position corresponding to the second user gesture in the display screen in the vehicle cabin and the gesture action corresponding to the second user gesture. By means of the scheme, gesture interaction of the multiple vehicle-mounted screens can be achieved, various different control logics are adopted according to different hand positions and gesture actions of the user so as to control the vehicle-mounted application program or the vehicle control component, the gesture control freedom degree is improved, and interaction between the user and the vehicle-mounted screens is more convenient.

Description

Technical Field

[0001] The present application relates to the field of vehicle control, and in particular to an interaction method, device, electronic device and storage medium based on a vehicle-mounted screen. Background Art

[0002] At present, there are more and more in-vehicle multimedia application ecosystems, and the interaction design with various display screens in the vehicle is becoming more and more rich and diverse. There are many types of multimedia applications based on the display screens in the vehicle.

[0003] Traditional vehicle interaction mainly relies on physical hardware such as buttons and knobs. Voice interaction is also a new interactive method that has been developed in recent years, making vehicle control and management more convenient. Vehicle gesture control is the latest way to control the vehicle infotainment system. As a supplement to other interactions, it is also playing an increasingly important role, with the advantages of easy operation and low learning cost. Summary of the invention

[0004] The embodiments of the present application provide an interactive method, device, electronic device and storage medium based on a vehicle-mounted screen to solve one or more of the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present application provides an interactive method for a vehicle-mounted screen, comprising:

[0006] recognizing a first user gesture through a high-definition camera disposed in the cockpit, and activating the gesture control mode when the first user gesture corresponds to activating the gesture control mode;

[0007] In gesture control mode, the spatial position of the user's hand in the car is mapped to the corresponding cursor position on the display screen in the cabin;

[0008] According to the current cursor position on the display screen in the vehicle cabin corresponding to the second user gesture and the gesture action corresponding to the second user gesture, the corresponding vehicle control instruction is executed.

[0009] In a second aspect, an embodiment of the present application provides an interactive device for a vehicle-mounted screen, comprising:

[0010] a control mode activation module, configured to recognize a first user gesture through a high-definition camera disposed in the cockpit, and activate the gesture control mode when the first user gesture corresponds to an activation gesture control mode;

[0011] A position mapping module, used to map the spatial position of the user's hand in the vehicle to the corresponding cursor position on the display screen in the vehicle cabin in the gesture control mode;

[0012] The instruction execution module is used to execute the corresponding vehicle control instruction according to the current cursor position corresponding to the second user gesture on the display screen in the vehicle cabin and the gesture action corresponding to the second user gesture.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0015] Compared with the related art, this application has the following advantages:

[0016] According to the embodiment of the present application, a high-definition camera arranged in the cabin is used to identify the first user's gesture. When the first user's gesture corresponds to the activation of the gesture control mode, the gesture control mode is started. Furthermore, when the gesture control mode is already turned on, the spatial position of the user's hand in the vehicle is mapped to the cursor position corresponding to the display screen in the vehicle cabin. Then, the corresponding vehicle control instruction can be executed according to the current cursor position corresponding to the display screen in the vehicle cabin of the second user's gesture and the gesture action corresponding to the second user's gesture. It can be seen that the embodiment of the present application realizes the operation of the display screen according to the user's gesture, and the program and various system functions in the display screen can be called through the user's gesture, such as controlling multimedia operations, including selecting films, switching between upper and lower sets, playing and pausing, etc.

[0017] Furthermore, when it is determined that the spatial position of the user's hand in the vehicle is mapped to the cursor position corresponding to the display screen in the cabin and the gesture action is determined to be about to exceed the display boundary of the currently mapped first display screen, a second display screen associated with the first display screen can be determined, and the spatial position of the user's hand in the vehicle collected in real time can be mapped to the cursor position corresponding to the second display screen. That is to say, by detecting that the cursor position mapped by the user's gesture exceeds the screen boundary, the operation range can be transferred from one display screen to another. When there are multiple display screens in the vehicle, the interactive screen boundary can be expanded, which is equivalent to realizing borderless screen content selection and control, and realizing the transfer and on-demand adjustment of the user's viewable screen.

[0018] Furthermore, according to the current cursor position corresponding to the second user gesture on the display screen in the vehicle cabin, the corresponding interactive area is determined to include switching controls between programs or function controls of system functions. The display screen can be switched to display multiple function controls of the target program after switching or a display window for adding function controls corresponding to system functions can be added. Then, according to the gesture action corresponding to the second user gesture, control instructions for the function controls of the target program after switching or the function controls corresponding to the system functions can be executed, thereby realizing switching from program use to calling of system functions by clicking specific controls within the program.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0021] Figure 1 A flow chart of an interactive method based on a vehicle-mounted screen provided in an embodiment of the present application is shown;

[0022] Figure 2 A flowchart of a gesture interaction method based on a vehicle-mounted screen provided in an embodiment of the present application is shown;

[0023] Figure 3a A schematic diagram showing screen display in a gesture interaction method provided in an embodiment of the present application is shown;

[0024] Figure 3b A schematic diagram showing screen display in another gesture interaction method provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of a software architecture solution of a gesture interaction method provided in an embodiment of the present application;

[0026] Figure 5 A structural block diagram of an interactive device based on a vehicle-mounted screen provided in an embodiment of the present application is shown; and

[0027] Figure 6 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0029] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0030] The embodiment of the present application provides an interactive method based on a vehicle screen, which is applied to a display screen in a smart car cockpit. A plurality of display screens are configured in the smart car cockpit, and the plurality of display screens can be adjacent in space or separated by a certain distance; the plurality of display screens can be based on different software and hardware facilities, such as connecting different processor chips or carrying different operating systems and software architectures.

[0031] The execution subject of the embodiment of the present application may be an application, service, instance, software-based functional module, virtual machine (VM), container or cloud server, etc., or a hardware device with data processing function (such as a server or terminal device) or hardware chip (such as CPU, GPU, FPGA, NPU, AI acceleration card or DPU), etc. The device for realizing vehicle information collection can be deployed on the computing device of the application party providing the corresponding service or on the cloud computing platform providing computing power, storage and network resources.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 Flow chart of an interactive method 100 based on a vehicle screen provided in an embodiment of the present application. Figure 1As shown, the method 100 may include the following steps S101-S103: Step S101, identifying a first user gesture through a high-definition camera arranged in the cockpit, and starting a gesture control mode when the first user gesture corresponds to activating the gesture control mode; Step S102, in the gesture control mode, mapping the spatial position of the user's hand in the vehicle to a cursor position corresponding to a display screen in the vehicle cabin; Step S103, executing a corresponding vehicle control instruction according to a current cursor position corresponding to a second user gesture on the display screen in the vehicle cabin and a gesture action corresponding to the second user gesture.

[0035] Next, the method 100 is described in detail step by step.

[0036] In step S101, a first user gesture is recognized by a high-definition camera arranged in the cabin, and when the first user gesture corresponds to activating the gesture control mode, the gesture control mode is started.

[0037] Gesture control is an important branch of the current in-vehicle intelligent control system. It relies on high-precision imaging devices such as high-definition cameras or optical detection devices to track and capture the user's hand movements in real time and accurately, as well as gesture recognition algorithms that can process the acquired photoelectric signals. Among them, imaging devices such as high-definition cameras or optical detection devices can be configured on the upper part of the in-vehicle display screen (equivalent to the front camera of the screen) or on the top of the cockpit. At present, the mainstream camera technologies for gesture recognition tasks include structured light technology, ToF (Time of Flight) and binocular ranging technology. Taking the ToF gesture recognition solution as an example, the ToF system is a light radar system that can emit light pulses from the emitter to the object, and the receiver can determine the distance of the measured object by calculating the running time of the light pulse from the emitter to the object and then returning to the receiver in pixel format. ToF gesture recognition uses the principle of ToF technology, that is, using the propagation characteristics of light in time and space to measure the distance, convert the distance into a time series signal, and then process the signal according to the time series, so as to achieve high-precision distance measurement and complete the recognition of gestures. The detection equipment, software or algorithm used in this application is not limited, as long as it can realize the gesture recognition function in this application.

[0038] When the display screen is running, even if the gesture control function mode is not entered, the system will continue to use the above-mentioned detection equipment to track, identify and judge the user's hand movement in real time, including the movement of the hand's spatial position and the hand action. For the first user gesture captured at a certain moment, when the system determines that the first user gesture is consistent with the preset gesture corresponding to the activation gesture control mode function, the gesture control mode is started, and only in the gesture control mode will the system respond to the user's hand movement. By setting a control gesture for activating the gesture control mode function, it can be avoided that the vehicle control command is mistakenly triggered due to the user's unintentional action being recognized by the system as a gesture indicating the execution of the vehicle control command. Among them, the display screen of the vehicle may include one or more of the central control screen, the co-pilot screen, the rear screen or other screens or media that can carry digital content.

[0039] The above is step S101, and then proceed to step S102.

[0040] In step S102, in the gesture control mode, the spatial position of the user's hand in the vehicle is mapped to a corresponding cursor position on the display screen in the vehicle cabin.

[0041] After step S101, the gesture control mode is activated. In the gesture control mode, the system will calculate the captured spatial position of the user's hand in the car in real time and convert the spatial position into the cursor position on the screen, that is, establish a mapping relationship between the spatial position of the user's hand in the car and the cursor position on the screen.

[0042] In some embodiments, in gesture control mode, mapping the spatial position of the user's hand in the car to the cursor position corresponding to the display screen in the cabin includes: in gesture control mode, based on the coordinate mapping relationship between the spatial coordinate system in the car and the screen coordinate system of the display screen, calculating the spatial position of the user's hand in the car and mapping it to the cursor position corresponding to the display screen in the cabin.

[0043] In order to project the captured user hand position (including its movement) on the display screen, the camera and other detection equipment in the cockpit can be calibrated in advance, and a three-dimensional space coordinate system in the vehicle can be established. The system models and digitizes the user's hand position and movement information (such as hand movement direction, speed or acceleration, etc.) in the vehicle's space coordinate system, and then calculates the user's hand position and movement information in the two-dimensional plane coordinate system of the display screen based on the pre-established coordinate conversion relationship between the three-dimensional space coordinate system in the vehicle and the two-dimensional plane coordinate system of the display screen. The conversion relationship between the three-dimensional space coordinate system in the vehicle and the two-dimensional plane coordinate system of the display screen can be adjusted manually or automatically during use. For example, if the user adjusts the sensitivity of gesture tracking or the cursor movement speed according to his own usage habits, the coordinate conversion relationship between the two coordinate systems in the system background algorithm will also change accordingly; or if the position parameters such as the orientation or tilt angle of the detection device or the display screen have changed, it is necessary to recalculate and determine the coordinate conversion relationship between the three-dimensional space coordinate system in the vehicle and the two-dimensional plane coordinate system of the display screen, and map the user's position movement to the corresponding cursor position on the screen according to the newly determined conversion relationship.

[0044] In some embodiments, the display screen includes multiple display screens, and the display screen coordinate system includes screen coordinate systems corresponding to the multiple display screens respectively. In gesture control mode, mapping the spatial position of the user's hand in the car to the cursor position corresponding to the display screen in the cabin also includes: when it is determined that the spatial position of the user's hand in the car is mapped to the cursor position corresponding to the display screen in the cabin and the gesture action is determined to be about to exceed the display boundary of the currently mapped first display screen, determining a second display screen associated with the first display screen, and mapping the spatial position of the user's hand in the car collected in real time to the second display screen.

[0045] In the case where the cockpit display screen includes multiple screens, the screen coordinate system of each display screen can be established respectively, and the coordinate conversion relationship between each screen coordinate system or the coordinate conversion relationship between each screen coordinate system and the three-dimensional coordinate system of the cockpit space can be established. When the system detects that the spatial position of the user's hand in the car is mapped to the cursor position corresponding to the display screen in the cabin and the gesture action is determined to exceed the first display screen where the currently mapped cursor is located, the second display screen associated with the first display screen is determined. The second display screen can be, for example, the screen closest to the spatial position of the first display screen and located in the moving direction of the cursor position corresponding to the spatial position of the user's hand in the car mapped to the display screen in the cabin; or the second display screen can also be a screen located on the opposite side of the first screen and the moving direction of the cursor position. For example, the cursor position moves to the right edge of the first display screen. When there is no display screen on the side of the first display screen on the moving direction of the cursor position, the cursor position beyond the right edge of the screen can be allowed to enter the second display screen on the left side of the first display screen from the left edge to improve the coherence of the user's gesture control of the cursor. When there are multiple display screens in the vehicle, the interactive screen boundaries can be expanded, which is equivalent to realizing borderless screen content selection and control, and enabling the transfer and on-demand adjustment of the user's viewing screen.

[0046] In some embodiments, in the gesture control mode, mapping the spatial position of the user's hand in the car to a cursor position corresponding to the display screen in the cabin also includes: displaying a gesture cursor at the cursor position, and the gesture cursor can move and display across the screen as the recognized movement of the user's hand is detected.

[0047] Furthermore, after the system maps the spatial position of the user's hand in the car to the corresponding cursor position on the screen, it can also display a gesture cursor at the position, which is used to indicate the position on the screen corresponding to the spatial position of the user's hand in the cockpit. According to the visual feedback of the gesture cursor on the movement of the user's hand, the convenience and accuracy of the user's gesture control can be improved. As described above, when the gesture cursor exceeds the edge of the first display screen as the user's hand moves in the cockpit space, the system will calculate and predict the corresponding cursor position on the second display screen associated with the first screen according to the parameters such as the moving direction, speed, and acceleration of the user's hand, and display the gesture cursor at the cursor position on the second display screen without interruption in the display. The cursor position on the second display screen can be directly calculated by the second system running on the chip or processor connected to the second display screen, or it can be calculated by the first system running on the chip or processor connected to the first display screen and sent to the second system corresponding to the second display screen, and then displayed on the screen by the second system. That is, in the embodiment of the present application, the calculation and display of the cursor position on the display screen can be two independent processes, and are implemented by independent software or hardware, which reduces the overall performance requirements or restrictions on the software and hardware equipment of the vehicle display screen.

[0048] In some embodiments, the number of display screens is greater than two, and determining the second display screen associated with the first display screen includes: determining multiple candidate display screens associated with the first display screen; obtaining position depth information of the user's hand and the multiple candidate display screens corresponding to the space inside the vehicle, and selecting a candidate display screen associated with the first display screen from the multiple candidate display screens as the second display screen.

[0049] The first display screen may have multiple candidate display screens associated with it, and the candidate display screens are display screens that are closest to the first display screen in spatial distance or are adjacent to or closely located to the first display screen. When the user's hand moves in the space of the vehicle cabin and the corresponding cursor position is about to exceed the display boundary of the first display screen, first determine the multiple candidate display screens associated with the first display screen, that is, the display screens that are adjacent to or close to the first display screen and located in the direction of the user's hand movement.

[0050] The candidate display screens determined above may include multiple ones, so it is necessary to further determine the second display screen from the multiple candidate display screens according to the depth information of the positions of the user's hand and the multiple candidate display screens in the vehicle interior space. The depth information of the positions of the user's hand and the multiple candidate display screens in the vehicle interior space is the longitudinal coordinate information of the positions of the user's hand and the multiple candidate display screens in the vehicle interior space.

[0051] In some embodiments, determining a plurality of candidate display screens associated with the first display screen includes: determining a plurality of candidate display screens associated with the first display screen from a preset display screen binding relationship.

[0052] In the case where the number of display screens in the cabin is greater than two, when the user's hand moves in the cabin space and the corresponding cursor position is about to exceed the display boundary of the first display screen, there may be multiple candidate display screens located in the direction of the user's hand movement. The candidate display screen has a preset display screen binding relationship with the first display screen, and the binding relationship can be set according to the spatial coordinates of each screen in the cabin. For example, display screens with adjacent or close spatial coordinates in the horizontal direction of the cabin are set to have a binding relationship and are mutually determined as candidate display screens associated with each other. The gesture cursor corresponding to the spatial position of the user's hand in the cabin is currently located on the first display screen on the instrument panel plane at the front of the cabin and is about to exceed the right boundary of the first display screen. There is a first candidate display screen on the instrument panel plane and on the right side of the first display screen. At the same time, there is a second candidate display screen on the right front seat backrest facing the rear seat, and the first candidate display screen and the second candidate display screen are close to the first display screen in the horizontal direction of the cabin. That is, there are two candidate display screens in the horizontal direction of the user's hand movement.

[0053] In some embodiments, selecting a candidate display screen associated with the first display screen from the multiple candidate display screens includes: selecting, from the multiple candidate display screens, a candidate display screen whose position depth information and the position depth information corresponding to the user's hand after performing a gesture action are the smallest, as the second display screen.

[0054] After determining multiple candidate display screens, it is necessary to further determine the second display screen based on the user's hand and each candidate display screen and the spatial depth information of the first display screen in the cabin. For example, the user performs gesture control on the first display screen at the front of the cabin in the back seat of the cabin. When the user's hand moves to the right side of the first display screen and is about to exceed the edge of the screen, the first candidate display screen on the right side of the first display screen and the second candidate display screen on the backrest of the right front seat facing the back seat are determined based on the binding relationship between the various screens in the horizontal direction. Next, based on the depth relationship between the current position of the user's hand and the position of the two candidate display screens, that is, the longitudinal distance between the user's hand and the first candidate display screen and the second candidate display screen in the cabin, the candidate display screen with the smaller distance is used as the second display screen. In this example, since the user is in the back row of the cabin, the longitudinal distance between his hand and the second candidate display screen is smaller. At this time, the system will determine that the cursor will enter the second candidate display screen after moving out of the first display screen and display the cursor.

[0055] The above is step S102, and then proceed to step S103.

[0056] In step S103, the corresponding vehicle control instruction is executed according to the current cursor position on the display screen in the vehicle cabin corresponding to the second user gesture and the gesture action corresponding to the second user gesture.

[0057] In some embodiments, the vehicle control instructions may include multimedia playback control instructions or vehicle hand-piece control instructions, and the media playback control instructions include multimedia playback, playback pause, volume control, selection of playback content, switching of playback content, or switching of multimedia playback screen. The vehicle hand-piece may include vehicle controlled components in or outside the cockpit, such as a vehicle display screen mechanical arm (used to adjust the tilt angle, extension / retraction distance, etc. of the vehicle screen), air conditioning, seat copying, door locks, etc.

[0058] It can be seen that according to the embodiment of the present application, the first user gesture is recognized by the high-definition camera set in the cockpit, and the gesture control mode is started when the first user gesture corresponds to the activation of the gesture control mode. Furthermore, when the gesture control mode has been turned on, the spatial position of the user's hand in the car is mapped to the cursor position corresponding to the display screen in the cabin, and then the corresponding vehicle control instruction can be executed according to the current cursor position corresponding to the display screen in the cabin of the second user gesture and the gesture action corresponding to the second user gesture. The embodiment of the present application realizes the operation of the display screen according to the user gesture, and the program and various system functions in the display screen can be called through the user gesture, such as controlling multimedia operations, including selecting movies, switching between upper and lower sets, playing and pausing, etc.

[0059] Figure 2 A flow chart of a gesture interaction method based on a vehicle-mounted screen provided in an embodiment of the present application is shown. Figure 2As shown, after the vehicle screen is turned on, it is in a waiting state and continues to track and identify the user's hand movements. When the user's hand movement is identified as a fist and an open palm for 1 second, the system activates the gesture control mode and prompts the user that the gesture control mode is successfully activated by displaying a dialog window or voice. Next, the system determines whether the interface opened on the current screen is a multimedia playback application. If not, the vehicle's counterpart is controlled by default, that is, according to the gesture action sent by the user next, the corresponding control instruction of the counterpart is determined and executed; if the current interface is a multimedia playback application, it enters the multimedia playback control, that is, according to the gesture action sent by the user next, the corresponding control instruction for multimedia playback is determined and executed, such as action 4: pinching and dragging left and right with two fingers to adjust the video progress bar, or action 6: grasping the palm to five fingers to exit the full screen video, etc. After the system executes the user's gesture corresponding instruction, it will continue to track and identify the user's gesture. When a fist gesture is recognized, or no other gestures are recognized within three seconds, the gesture control is terminated; if the system recognizes a gesture other than a fist within three seconds, the steps of judging the current interface and then executing the corresponding instruction according to the interface are repeated.

[0060] In some embodiments, the display screen is divided into a plurality of interactive areas corresponding to different control objects, the control objects include programs or corresponding functional controls within the programs, and the programs include multimedia playback programs or vehicle-mounted system applications.

[0061] On one or more screens, a plurality of different control objects may be displayed simultaneously, and the screen may be divided into a plurality of different interactive areas according to the plurality of different control objects. Figure 3a A screen diagram of a vehicle screen interaction method provided in an embodiment of the present application, such as Figure 3aAs shown, the first application interface is displayed in the first screen 310 on the left, the first application is a multimedia application, and the current interface is a movie selection interface; the second application interface is displayed in the second screen 320 on the right, and the second application can be different from the first application. The first application and the second application are two different control objects, and the first area 311 and the second area 321 respectively occupied on the screen are the corresponding interaction areas; at the same time, the control object and the corresponding control area can be further divided in the first interaction area. For example, a plurality of movies to choose from are displayed in the first interaction area 311, and the icon of each movie is also a control object for gesture control, and occupies different interaction areas accordingly. That is, the control object in the embodiment of the present application can be either an application interface or a control, UI element, etc. in the application interface (including the operating system interface), such as an application window or a video stream. In the embodiment of the present application, the screen cursor position corresponding to the user's hand position is located in different interaction areas, which will cause the system to select different gesture judgment logics to determine the corresponding vehicle control instructions, that is, the same gesture action has different corresponding vehicle control instructions when the cursor position is in the interaction area corresponding to different control objects.

[0062] In a possible implementation, when executing the corresponding vehicle control command according to the current cursor position on the display screen corresponding to the second user gesture in the vehicle cabin and the gesture action corresponding to the second user gesture, specifically, firstly, the corresponding interactive area can be determined according to the current cursor position on the display screen corresponding to the second user gesture in the vehicle cabin, and then, according to the gesture action corresponding to the second user gesture, the corresponding control command for the control object in the interactive area is executed. The correspondence between the gesture action and the control command can be configured according to actual needs.

[0063] In a possible implementation, the control object further includes a switching control between programs, the switching control is used to trigger the switching between one program and another program, and when the gesture action corresponding to the user gesture indicates triggering the switching control, the switching logic corresponding to the switching control can be executed to switch from the currently displayed program to another program. Alternatively, the control object can also include a function control of a system function, the function control is used to display the system function, and when the gesture action corresponding to the user gesture indicates triggering the function control, the function logic corresponding to the function control can be executed, that is, the control instruction corresponding to the system function is executed. When determining the corresponding interactive area according to the current cursor position corresponding to the display screen of the second user gesture in the cabin, it can be determined according to the current cursor position corresponding to the display screen of the second user gesture in the cabin that the corresponding interactive area includes the switching control between programs or the function control of the system function, that is, it is determined that the current position corresponding to the second user gesture on the display screen has moved to the interactive area of ​​the display screen displaying the switching control between programs or the function control of the system function, and further the display screen can be switched from the original display content to display multiple function controls of the target program after the switch or increase the display window of the function control corresponding to the system function. Furthermore, when the corresponding control instructions for the control object of the interactive area are executed according to the gesture action corresponding to the second user gesture, that is, the control instructions for the function controls of the switched target program or the function controls corresponding to the system functions are executed according to the gesture action corresponding to the second user gesture, thereby enabling switching from program use to calling of system functions by clicking on specific controls within the program.

[0064] Figure 3b FIG. 1 is a schematic diagram showing screen display in another gesture interaction method provided in an embodiment of the present application. Figure 3b As shown, the third application (video playback application) is displayed in full screen on the right screen, that is, the right screen plays the video in full screen at this time, and area 350 is the interactive area of ​​the third application. When the user moves his hand to make the gesture cursor 330 on the corresponding display screen move to the preset position (the left edge of the right screen in the figure), the system will automatically display an interactive object, which corresponds to the fourth application. The interactive area 340 of the interactive object does not belong to the interactive area 350. When the gesture cursor is located in the interactive area 340, the system will select the gesture control logic of the fourth application in the interactive area 340 according to the received user gesture action, and will no longer execute the video playback gesture control logic corresponding to the third application. That is, by sensing and locating the gesture cursor position corresponding to the current hand position of the user, the interactive area of ​​the interactive object that was not originally displayed on the screen can be additionally displayed on the screen to interact with the interactive object, thereby improving the user's freedom of interactive control of the display screen.

[0065] Figure 4 The software architecture scheme of a gesture interaction method provided in an embodiment of the present application is schematically shown, which is applied to the gesture control scenario of the main and auxiliary display screens in the front row of the vehicle cabin. Figure 4As shown in the figure, the software architecture levels of the main and auxiliary display screens are consistent. The dotted arrows in the figure represent the video data flow path, and the solid arrows represent the control command flow path. As shown in the dotted path in the figure, the vehicle camera sends the video data to the CameraService module located in the basic platform layer. The IdentifyService module located in the basic platform layer periodically obtains the video data from the CameraService module, and then calculates the gesture space position and specific gesture action corresponding to the video data through its internally integrated gesture algorithm submodule, and synchronizes the gesture space position and gesture action to the IntelligentInput module located in the basic service layer; the IntelligentInput module also defines the coordinate ranges corresponding to the main and auxiliary display screens respectively. It determines the position of the gesture position on the display screen according to the gesture space position sent by IdentifyService, so as to determine whether the instruction corresponding to the gesture action is sent to the main display screen or the auxiliary display screen. When sending the instruction to the main display screen, it is necessary to further determine the target functional service module for sending the instruction according to the foreground application of the current main display screen. For example, path 1 in the figure indicates that the IntelligentInput module determines that the current gesture spatial position corresponds to the main display screen. Next, the IntelligentInput module obtains the current main screen foreground application information from the SrcnMgrCenter module. When it is determined that the current foreground page is a multimedia application, the gesture action corresponding instruction is sent to the multimedia application on the main display screen so that the latter can perform multimedia playback control operations; path 2 in the figure indicates that when the IntelligentInput module obtains the current main screen foreground application information from the SrcnMgrCenter module, and determines that the current foreground page is a hand control program, such as a robotic arm control program, by calling the information in the SrcnMgrCenter module. sequence, the corresponding instructions for controlling the opponent are sent to the RacService module located in the basic platform layer so that the latter calls the corresponding hardware driver interface to control the movement of the robotic arm; as another example shown in path 3, when the IntelligentInput module determines that the current gesture spatial position corresponds to the secondary display screen, the gesture action will be sent to the GestureMngCenter module of the secondary display screen through the OpenGW module, wherein the OpenGW module is used for protocol conversion and parsing of data transmission between the main and secondary display screens, and the GestureMngCenter module sends the corresponding instructions of the gesture action to the multimedia application of the secondary display screen for multimedia playback control operations according to the current foreground application (multimedia application) of the current secondary display screen. It should be noted that Figure 4 And the above correspondence Figure 4The related descriptions have made certain simplifications on the software architecture and implementation steps, and their purpose is only to serve as examples to illustrate the interaction logic of each functional module in the gesture interaction method provided in the embodiment of the present application, and is not intended to limit the present application.

[0066] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a control device for a vehicle motor. Figure 5 The structural block diagram of an interactive device 500 based on a vehicle-mounted screen provided in an embodiment of the present application, the device 500 may include:

[0067] A control mode starting module 501 is used to recognize a first user gesture through a high-definition camera provided in the cockpit, and start the gesture control mode when the first user gesture corresponds to an activation gesture control mode;

[0068] A position mapping module 502, used to map the spatial position of the user's hand in the vehicle to a corresponding cursor position on a display screen in the vehicle cabin in the gesture control mode;

[0069] The instruction execution module 503 is used to execute the corresponding vehicle control instruction according to the current cursor position corresponding to the second user gesture on the display screen in the vehicle cabin and the gesture action corresponding to the second user gesture.

[0070] In one possible implementation, the position mapping module 502 is specifically used to calculate the spatial position of the user's hand in the car and map it to the cursor position corresponding to the display screen in the cabin according to the coordinate mapping relationship between the spatial coordinate system in the car and the screen coordinate system of the display screen in the gesture control mode.

[0071] In one possible implementation, the display screen includes multiple display screens, and the display screen coordinate system includes screen coordinate systems corresponding to the multiple display screens respectively; the position mapping module 502 is also specifically used to determine the second display screen associated with the first display screen when it is determined that the spatial position of the user's hand in the car is mapped to the cursor position corresponding to the display screen in the cabin and the gesture action is determined to be about to exceed the display boundary of the currently mapped first display screen, and map the spatial position of the user's hand in the car collected in real time to the cursor position corresponding to the second display screen.

[0072] In a possible implementation, a gesture cursor is displayed at the cursor position, and the gesture cursor can be moved and displayed across the screen along with the recognized movement of the user's hand.

[0073] In a possible implementation, the number of the display screens is greater than two, and the position mapping module 502 includes:

[0074] a candidate display screen determination submodule, configured to determine a plurality of candidate display screens associated with the first display screen;

[0075] The second display screen determination submodule is used to obtain the position depth information of the user's hand and the multiple candidate display screens respectively corresponding to the space inside the vehicle, and select a candidate display screen associated with the first display screen from the multiple candidate display screens as the second display screen.

[0076] In a possible implementation, the candidate display screen determination submodule is further specifically configured to determine a plurality of candidate display screens associated with the first display screen from a preset display screen binding relationship.

[0077] In one possible implementation, the second display screen determination submodule is further specifically used to select, from the multiple candidate display screens, a candidate display screen whose position depth information and the position depth information corresponding to the user's hand after performing the gesture action are the smallest, as the second display screen.

[0078] In one possible implementation, the vehicle control instructions include multimedia playback control instructions or vehicle hand control instructions, and the media playback control instructions include multimedia playback, playback pause, volume control, selection of playback content, switching of playback content, or switching of multimedia playback screens.

[0079] In a possible implementation, the display screen is divided into a plurality of interactive areas corresponding to different control objects, the control objects include programs or corresponding functional controls within the programs, and the programs include multimedia playback programs or vehicle-mounted system applications.

[0080] In a possible implementation, the instruction execution module 503 includes:

[0081] An interaction area determination submodule, configured to determine a corresponding interaction area according to a current cursor position corresponding to the display screen in the vehicle cabin of the second user gesture;

[0082] The instruction execution submodule is used to execute the corresponding control instruction on the control object of the interactive area according to the gesture action corresponding to the second user gesture.

[0083] In a possible implementation, the control object further includes a switching control between programs or a function control of a system function, and the interaction area determination submodule includes:

[0084] a space determination subunit, configured to determine, according to a current cursor position corresponding to the display screen in the vehicle cabin of the second user gesture, a corresponding interactive area including a switching control between programs or a function control of a system function;

[0085] A display window switching subunit, used to switch the display screen to display multiple function controls of the switched target program or to add a display window for displaying function controls corresponding to system functions;

[0086] The instruction execution submodule 503 is specifically used to execute the control instruction of the function control of the switched target program or the function control corresponding to the system function according to the gesture action corresponding to the second user gesture.

[0087] According to the embodiment of the present application, a high-definition camera arranged in the cabin is used to identify the first user's gesture. When the first user's gesture corresponds to the activation of the gesture control mode, the gesture control mode is started. Furthermore, when the gesture control mode is already turned on, the spatial position of the user's hand in the vehicle is mapped to the cursor position corresponding to the display screen in the vehicle cabin. Then, the corresponding vehicle control instruction can be executed according to the current cursor position corresponding to the display screen in the vehicle cabin of the second user's gesture and the gesture action corresponding to the second user's gesture. It can be seen that the embodiment of the present application realizes the operation of the display screen according to the user's gesture, and the program and various system functions in the display screen can be called through the user's gesture, such as controlling multimedia operations, including selecting films, switching between upper and lower sets, playing and pausing, etc.

[0088] Furthermore, when it is determined that the spatial position of the user's hand in the vehicle is mapped to the cursor position corresponding to the display screen in the cabin and the gesture action is determined to be about to exceed the display boundary of the currently mapped first display screen, a second display screen associated with the first display screen can be determined, and the spatial position of the user's hand in the vehicle collected in real time can be mapped to the cursor position corresponding to the second display screen. That is to say, by detecting that the cursor position mapped by the user's gesture exceeds the screen boundary, the operation range can be transferred from one display screen to another. When there are multiple display screens in the vehicle, the interactive screen boundary can be expanded, which is equivalent to realizing borderless screen content selection and control, and realizing the transfer and on-demand adjustment of the user's viewable screen.

[0089] Furthermore, according to the current cursor position corresponding to the second user gesture on the display screen in the vehicle cabin, the corresponding interactive area is determined to include switching controls between programs or function controls of system functions. The display screen can be switched to display multiple function controls of the target program after switching or a display window for adding function controls corresponding to system functions can be added. Then, according to the gesture action corresponding to the second user gesture, control instructions for the function controls of the target program after switching or the function controls corresponding to the system functions can be executed, thereby realizing switching from program use to calling of system functions by clicking specific controls within the program.

[0090] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0091] Figure 6 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 601 and a processor 602. The memory 601 stores a computer program that can be run on the processor 602. When the processor 602 executes the computer program, the method in the above embodiment is implemented. The number of the memory 601 and the processor 602 can be one or more.

[0092] The electronic device also includes:

[0093] The communication interface 603 is used to communicate with external devices and perform data exchange transmission.

[0094] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the memory 601, the processor 602 and the communication interface 603 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0095] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0096] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0097] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented.

[0098] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0099] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0100] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0101] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0105] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0106] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0107] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0108] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0109] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An interactive method based on a vehicle screen, comprising: recognizing a first user gesture through a high-definition camera disposed in the cockpit, and activating the gesture control mode when the first user gesture corresponds to activating the gesture control mode; In gesture control mode, the spatial position of the user's hand in the car is mapped to the corresponding cursor position on the display screen in the cabin; According to the current cursor position on the display screen in the vehicle cabin corresponding to the second user gesture and the gesture action corresponding to the second user gesture, the corresponding vehicle control instruction is executed.

2. The method according to claim 1, wherein: In the gesture control mode, mapping the spatial position of the user's hand in the vehicle to the cursor position corresponding to the display screen in the vehicle cabin includes: In gesture control mode, based on the coordinate mapping relationship between the in-car spatial coordinate system and the screen coordinate system of the display screen, the spatial position of the user's hand in the car is calculated and mapped to the cursor position corresponding to the display screen in the cabin.

3. The method according to claim 2, wherein: The display screen includes a plurality of display screens, and the display screen coordinate system includes screen coordinate systems corresponding to the plurality of display screens respectively; In the gesture control mode, mapping the spatial position of the user's hand in the vehicle to the corresponding cursor position on the display screen in the vehicle cabin also includes: When it is determined that the spatial position of the user's hand in the car is mapped to the cursor position corresponding to the display screen in the cabin, and the gesture action is determined to be about to exceed the display boundary of the currently mapped first display screen, a second display screen associated with the first display screen is determined, and the spatial position of the user's hand in the car collected in real time is mapped to the second display screen.

4. The method according to claim 2, wherein: A gesture cursor is displayed at the cursor position, and the gesture cursor moves and is displayed across the screen along with the recognized movement of the user's hand.

5. The method according to claim 3, wherein: The number of the display screens is greater than two, and determining the second display screen associated with the first display screen comprises: determining a plurality of candidate display screens associated with the first display screen; The depth information of the positions of the user's hand and the multiple candidate display screens respectively corresponding to the space inside the vehicle is obtained, and a candidate display screen associated with the first display screen is selected from the multiple candidate display screens as the second display screen.

6. The method according to claim 5, wherein: The determining a plurality of candidate display screens associated with the first display screen comprises: A plurality of candidate display screens associated with the first display screen are determined from preset display screen binding relationships.

7. The method according to claim 5, wherein: The selecting a candidate display screen associated with the first display screen from the plurality of candidate display screens as the second display screen comprises: A candidate display screen whose position depth information and the position depth information corresponding to the user's hand after the gesture action are performed are the smallest is selected from the multiple candidate display screens as the second display screen.

8. The method according to claim 1, wherein: The vehicle control instructions include multimedia playback control instructions or vehicle handpiece control instructions, and the media playback control instructions include multimedia playback, playback pause, volume control, selection of playback content, switching of playback content or switching of multimedia playback screens.

9. The method according to claim 1, wherein: The display screen is divided into a plurality of interactive areas corresponding to different control objects, wherein the control objects include programs or corresponding functional controls within the programs, and the programs include multimedia playback programs or vehicle-mounted systems.

10. The method according to claim 9, wherein: The executing the corresponding vehicle control instruction according to the current cursor position on the display screen corresponding to the second user gesture in the vehicle cabin and the gesture action corresponding to the second user gesture comprises: Determining a corresponding interaction area according to a current cursor position on a display screen in the vehicle cabin corresponding to the second user gesture; According to the gesture action corresponding to the second user gesture, a corresponding control instruction is executed on the control object of the interaction area.

11. The method according to claim 10, wherein: The control object also includes a switching control between programs or a function control of a system function, and the determining of the corresponding interactive area according to the current cursor position corresponding to the display screen in the vehicle cabin according to the second user gesture includes: Determining, according to the current cursor position corresponding to the display screen in the vehicle cabin of the second user gesture, that the corresponding interactive area includes a switching control between programs or a function control of a system function; Switching the display screen to display multiple function controls of the switched target program or adding a display window for displaying function controls corresponding to system functions; The executing a corresponding control instruction on the control object of the interactive area according to the gesture action corresponding to the second user gesture includes: According to the gesture action corresponding to the second user gesture, a control instruction for the function control of the switched target program or the function control corresponding to the system function is executed.

12. An interactive device based on a vehicle-mounted screen, comprising: a control mode activation module, configured to recognize a first user gesture through a high-definition camera disposed in the cockpit, and activate the gesture control mode when the first user gesture corresponds to an activation gesture control mode; A position mapping module, used to map the spatial position of the user's hand in the vehicle to the corresponding cursor position on the display screen in the vehicle cabin in the gesture control mode; The instruction execution module is used to execute the corresponding vehicle control instruction according to the current cursor position corresponding to the second user gesture on the display screen in the vehicle cabin and the gesture action corresponding to the second user gesture.

13. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images