A method, device, terminal device and vehicle for multi-screen interaction

Abstract

The application provides a multi-screen interaction method and device, terminal equipment and vehicle, and relates to the technical field of intelligent vehicles. After detecting that a specific gesture is included in an image captured by a camera, the content displayed on a first display screen at this time is converted into a sub-image, and the identification of other display screens capable of displaying the sub-image and the orientation information relative to the first display screen are displayed on the first display screen. The user can intuitively see the moving direction of the subsequent gesture, and then determines a second display screen according to the moving direction of the specific gesture, and controls the sub-image to move on the first display screen along with the movement of the specific gesture. When it is monitored that the distance of the movement of the specific gesture is greater than a set threshold, the sub-image is moved to the second display screen, thereby realizing the multi-screen interaction function.

Description

[0001] This application is a divisional application. The original application has the application number 202180001484.X and the original application date is April 26, 2021. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent vehicles, and more particularly to a method, apparatus, terminal device, and vehicle for multi-screen interaction. Background Technology

[0003] With the development of intelligent technology, vehicles are becoming increasingly intelligent, and the functions of smart cockpits are attracting more and more attention from consumers. To enhance the driver's experience, vehicles have multiple displays: one for information such as speed, fuel consumption, and mileage; another for navigation routes; and yet another for entertainment videos such as music and radio, as well as other functions. Typically, each display shows fixed content and cannot show content intended for other displays. However, with the development of technologies such as screen mirroring and screen sharing, enabling one display to push its content to other displays for shared viewing has become a key user requirement and a research hotspot for manufacturers.

[0004] For vehicles equipped with multiple displays, in the existing technology, if the driver wants to push the content displayed on one display to another, the driver needs to click a specific button on the display or drag the target content on the display. This method of requiring the driver to operate on the display distracts the driver and poses a safety problem when driving the vehicle. Summary of the Invention

[0005] To address the aforementioned issues, embodiments of this application provide a method, apparatus, terminal device, and vehicle for multi-screen interaction.

[0006] In a first aspect, this application provides a method for multi-screen interaction, comprising: acquiring sensing information, the sensing information including gesture information; triggering a first display screen to display a first interface image based on the gesture information, the first interface image including a sub-image, the movement trend of the sub-image being associated with the gesture information; and triggering the sub-image to be displayed on a second display screen.

[0007] In this embodiment, sensor information is acquired to obtain gesture information. Then, based on the gesture information, the display screen is triggered to display an interface image and a sub-image is displayed. By associating the sub-image with the gesture information, the sub-image moves with the movement of the gesture. By triggering certain conditions, such as the movement distance of the gesture being greater than a set threshold, the sub-image is triggered to move to another display screen, thereby realizing the multi-screen interaction function.

[0008] In one implementation, when the gesture information includes a five-finger grasping gesture, the content displayed by the sub-image is the entire content of the first interface image.

[0009] In one implementation, when the gesture information includes a two-finger grasping gesture, the content displayed in the sub-image is the interface presented by the first application in the first interface image, where the first application is an application selected by the user or an application that is currently running.

[0010] In one embodiment, the first interface image further includes location information, which is an identifier of at least one other display screen that can display the sub-image, and orientation information of the at least one display screen relative to the first display screen.

[0011] In this embodiment, by displaying the identifiers of other displays capable of displaying sub-images and their orientation information relative to the first display screen, users can intuitively see the direction of movement of subsequent gestures, facilitating subsequent multi-screen interaction.

[0012] In one embodiment, before triggering the display of the sub-image on the second display screen, the method further includes: determining the second display screen, wherein the second display screen is determined based on first position information, second position information and stored orientation information of at least one display screen relative to the first display screen, the first position information being the position information of the gesture when triggering the first display screen to display the first interface image, the second position information being the position information of the gesture at the current moment, and the at least one display screen including the second display screen.

[0013] In this implementation, by pre-determining the target display screen for interaction, subsequent sub-images can be quickly moved to the target display screen when triggered, reducing processing time and improving user experience.

[0014] In one embodiment, determining the second display screen includes: determining first orientation information of the second location information relative to the first location information based on the first location information and the second location information; comparing the first orientation information with the orientation information of at least one stored display screen relative to the first display screen; and determining the second display screen when the first orientation information is the same as the orientation information of the second display screen relative to the first display screen.

[0015] In this embodiment, by associating the directional information of gestures with the directional information between displays, users only need to move in space to generate directional changes to achieve multi-screen interaction, making the operation of this function simple and easy to implement.

[0016] In one embodiment, triggering the display of the sub-image on the second display screen includes: when the distance between the first location information and the second location information is detected to be greater than a set threshold, triggering the display of the sub-image on the second display screen.

[0017] In this implementation, by associating the distance of gesture movement with multi-screen interaction, users only need to move in space to create a large distance difference to achieve multi-screen interaction, making the operation of this function simple and easy to implement.

[0018] In one embodiment, when the sub-image is an application icon, the triggering of displaying the sub-image on the second display screen includes: triggering the image indicated by the sub-image to be displayed on the second display screen.

[0019] In this implementation, if the sub-image is an image of an application, after the sub-image is moved to the target display screen, the application is run directly without user intervention, so as to make it more convenient for the user. The user is not required to actively operate on the display screen to run the application.

[0020] In one embodiment, the size of the sub-image is smaller than the size of the first display screen to avoid the sub-image completely covering the original interface image and affecting the user's viewing of the original interface image content.

[0021] In one embodiment, the location information is displayed at the edge of the first display screen so that the user can more intuitively see the relative positions of other display screens.

[0022] In one embodiment, when the resolution of the first display screen is different from the resolution of the second display screen, the method further includes: setting the resolution of the sub-image to the resolution of the second display screen to avoid the sub-image being unable to be displayed on the target display screen due to pixel issues.

[0023] In one embodiment, when the size of the first display screen is different from the size of the second display screen, the method further includes: setting the size of the sub-image to the size of the second display screen to avoid the sub-image being unable to be displayed on the target display screen due to size issues.

[0024] In one embodiment, when the size of the long side or the size of the short side of the second display screen is smaller than that of the first display screen, the method further includes: reducing the sub-image to the same size as the long side of the second display screen; or reducing the sub-image to the same size as the short side of the second display screen.

[0025] In this embodiment, if the aspect ratio of the sub-image is different from that of the target display screen, the long side of the sub-image can be aligned with the target display screen, or the short side of the sub-image can be aligned with the target display screen, so that the sub-image can be displayed on the display screen.

[0026] Secondly, embodiments of this application provide a multi-screen interaction device, comprising: a transceiver unit for acquiring sensing information, the sensing information including gesture information; a processing unit for triggering a first display screen to display a first interface image based on the gesture information, the first interface image including sub-images, the movement trend of the sub-images being associated with the gesture information; and triggering the sub-images to be displayed on a second display screen.

[0027] In one implementation, when the gesture information includes a five-finger grasping gesture, the content displayed by the sub-image is the entire content of the first interface image.

[0028] In one implementation, when the gesture information includes a two-finger grasping gesture, the content displayed in the sub-image is the interface presented by the first application in the first interface image, where the first application is an application selected by the user or an application that is currently running.

[0029] In one embodiment, the first interface image further includes location information, which is an identifier of at least one other display screen that can display the sub-image, and orientation information of the at least one display screen relative to the first display screen.

[0030] In one embodiment, the processing unit is further configured to determine the second display screen, the second display screen being determined based on first location information, second location information and stored orientation information of at least one display screen relative to the first display screen, the first location information being the location information of the gesture when the first display screen is triggered to display the first interface image, the second location information being the location information of the gesture at the current moment, and the at least one display screen including the second display screen.

[0031] In one embodiment, the processing unit is specifically configured to: determine first orientation information of the second location information relative to the first location information based on the first location information and the second location information; compare the first orientation information with the orientation information of at least one stored display screen relative to the first display screen; and determine the second display screen when the first orientation information is the same as the orientation information of the second display screen relative to the first display screen.

[0032] In one embodiment, the processing unit is specifically used to trigger the display of the sub-image on the second display screen when it detects that the distance between the first location information and the second location information is greater than a set threshold.

[0033] In one embodiment, when the sub-image is an application icon, the processing unit is specifically configured to trigger the display of the image indicated by the sub-image on the second display screen.

[0034] In one embodiment, the size of the sub-image is smaller than the size of the first display screen.

[0035] In one embodiment, the location information is displayed at the edge of the first display screen.

[0036] In one embodiment, when the resolution of the first display screen is different from the resolution of the second display screen, the processing unit is further configured to set the resolution of the sub-image to the resolution of the second display screen.

[0037] In one embodiment, when the size of the first display screen is different from the size of the second display screen, the processing unit is further configured to set the size of the sub-image to the size of the second display screen.

[0038] In one embodiment, when the size of the long side or the size of the short side of the second display screen is smaller than that of the first display screen, the processing unit is further configured to reduce the sub-image to the same size as the long side of the second display screen; or reduce the sub-image to the same size as the short side of the second display screen.

[0039] Thirdly, embodiments of this application provide a multi-screen interactive system, including: a processor and at least two displays, for executing various possible implementations of the first aspect.

[0040] Fourthly, embodiments of this application provide a vehicle including: at least one camera, at least two displays, at least one memory, and at least one processor for executing the various possible implementations of the first aspect.

[0041] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the various possible implementations of the first aspect.

[0042] In a sixth aspect, embodiments of this application provide a computing device, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the various possible implementations of the first aspect.

[0043] In a seventh aspect, a computing device is provided, the computing device comprising: a processor and an interface circuit; wherein the processor is coupled to a memory via the interface circuit, the processor being used to execute program code in the memory to implement the technical solutions provided by any one of the second to fourth aspects or any possible embodiments.

[0044] In this embodiment, when a driver wants to push content displayed on one screen to another, to avoid safety hazards caused by the driver clicking specific buttons or dragging target content on the screen, this application only requires the driver to naturally place their hand next to the source screen and open their palm. After the activation gesture is recognized, the source screen enters the flying screen activation state as the fingers gradually close. The whole process is equivalent to the hand performing a grabbing operation on the current screen. Then, the driver only needs to move in the corresponding direction according to the interface prompts. During the movement, the driver can also see in real time which direction the source screen is flying. After the movement distance reaches a threshold, the fingers open to perform the flying screen operation. The whole process is equivalent to grabbing the source screen and throwing it to the target screen. The operation is simple and does not require much effort from the driver to achieve multi-screen interaction, thereby ensuring the safety of the vehicle during driving. Attached Figure Description

[0045] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0046] Figure 1 A schematic diagram of the structure of a vehicle provided in an embodiment of this application;

[0047] Figure 2(a) is a schematic diagram showing the location of the display screen and camera in the front seat of a vehicle according to an embodiment of this application;

[0048] Figure 2(b) is a schematic diagram showing the location of the display screen and camera in the rear seat of a vehicle according to an embodiment of this application;

[0049] Figure 3 This is a schematic diagram illustrating the implementation process of a multi-screen interaction method provided in an embodiment of this application;

[0050] Figure 4(a) is a schematic diagram of gesture changes for implementing five-finger grasping to achieve screen projection function according to an embodiment of this application;

[0051] Figure 4(b) is a schematic diagram of gesture changes for implementing two-finger grasping and application sharing functions according to an embodiment of this application;

[0052] Figure 5 A schematic diagram illustrating the recognition of gestures in an image, provided as an embodiment of this application;

[0053] Figure 6 This application provides a schematic diagram of the source display screen image and corresponding gesture when acquiring a sub-image during the process of implementing a five-finger grasp to achieve the display screen projection function;

[0054] Figure 7 This application provides an embodiment of a source display screen showing the image and corresponding gesture when acquiring a sub-image during the process of implementing a two-finger grasp to achieve application sharing functionality;

[0055] Figure 8 This application provides a schematic diagram illustrating the display effect after a portion of the image on the source display screen is removed, as part of an embodiment of the present application.

[0056] Figure 9 This is a schematic diagram illustrating the display effect of a partial sub-image after being moved into a target display screen, as provided in an embodiment of this application.

[0057] Figure 10 One of the schematic diagrams illustrating the display effect of a sub-image on a target display screen provided in an embodiment of this application;

[0058] Figure 11 A second schematic diagram illustrating the display effect of a sub-image on a target display screen provided in an embodiment of this application;

[0059] Figure 12 The third schematic diagram illustrating the display effect of the target display screen showing the sub-image in the embodiment of this application;

[0060] Figure 13 This application provides an embodiment of a schematic diagram illustrating the movement effect of two sub-images when a source display screen simultaneously shares data with two target display screens.

[0061] Figure 14 A schematic diagram illustrating a five-finger grasping gesture and sub-image movement process for implementing screen projection functionality, provided in an embodiment of this application;

[0062] Figure 15 A schematic diagram of a multi-screen interaction device structure provided in an embodiment of this application;

[0063] Figure 16 This is a schematic diagram of another multi-screen interaction device provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0065] Figure 1 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. Figure 1 As shown, the vehicle 100 includes at least two displays 101 (e.g., Figure 1 The displays 101-1 to 101-N shown), and at least one camera 102 (e.g., Figure 1 The diagram shows cameras 102-1 to 102-M, a processor 103, a memory 104, and a bus 105. The display 101, cameras 102, processor 103, and memory 104 can establish a communication connection via the bus 105. Here, N is an integer greater than 1, and M is a positive integer.

[0066] The display screen 101 may include one or more of the following: a touch screen display and a non-touch screen display. For example, display screen 101 may be entirely touch screen displays, entirely non-touch screen displays, or it may include both types. Display screen 101 can be used to display instrument data such as fuel level, vehicle speed, and mileage, as well as navigation routes, music, videos, and images (such as images of the vehicle's surroundings). To save costs, a non-touch screen can be used to display instrument data such as fuel level, vehicle speed, and mileage, while a touch screen can display navigation routes, music, videos, and images. It should be understood that a touch screen display can also be used to display instrument data such as fuel level, vehicle speed, and mileage. The above display content is merely an example, and this application does not limit the content displayed on the display screen.

[0067] For example, the position of the display screen 101 can be set with reference to Figure 2(a) or Figure 2(b). In Figure 2(a), the front row of the vehicle 100 can have a display screen 101-1 positioned in the center of the steering wheel, displaying functions such as volume buttons for music playback, play / pause buttons, and answer / hang-up buttons, facilitating easier operation for the driver while driving, reducing the need for the driver to switch between lines of sight and body angles, and improving driving safety. A display screen 101-2 is installed on the vehicle body below the windshield and above the steering wheel, displaying data such as fuel level, speed, and mileage; a display screen 101-3 is installed between the driver's and passenger's seats, displaying data such as music, videos, and navigation routes; and for the passenger's seat, a display screen 101-4 can be installed on the vehicle body in front of the passenger seat, displaying any content the passenger in the passenger seat wishes to view. For the rear seats of the vehicle 100 shown in Figure 2(b), a display screen 101-5 and a display screen 101-6 can be installed above the back of each front seat. These two displays screens can display any content that the rear passengers want to watch, such as movies, navigation information, weather, etc.

[0068] The number and location of the display screens 101 in this embodiment are not limited to the number and location relationships shown in Figures 2(a) and 2(b). This application is merely illustrative to help readers understand the solution. The specific number and location of the display screens 101 on the vehicle 100 are determined according to the actual situation.

[0069] Camera 102 can be used to capture still images or videos. For example, an object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to a processor (such as an image processor) to be converted into a digital image signal. The digital image signal is output to a digital signal processor (DSP) for processing. The DSP converts the digital image signal into standard red-green-blue (RGB) and luminance-bandwidth-chrominance (YUV) format image signals. Camera 102 can be installed in different locations inside the vehicle. Inside the vehicle, camera 102 can be used to collect limb information (e.g., palm information) of the user inside the vehicle 100. Camera 102 can operate by periodically taking pictures or continuously capturing video streams. This application will subsequently describe the technical solution of this application using periodic picture taking as an example.

[0070] In this application, the positions of the various cameras 102 on the vehicle 100 are shown in Figures 2(a) and 2(b). For the front seats, a camera 102-1 can be installed on the rearview mirror, with its shooting range covering the driver's seat and the front passenger seat, to collect the palm information of the users in the driver's seat and the front passenger seat; for the rear seats, cameras 102-3 and 102-4 can be installed above the displays 101-5 and 101-6 respectively, and cameras 102-2 and 102-5 can be installed on the two rear door frames near the front seats respectively, to collect the palm information of the users in the rear seats.

[0071] Similarly, the number and location of the cameras 102 in this embodiment are not limited to the number and location relationships shown in Figures 2(a) and 2(b). This application is merely illustrative to help readers understand the solution. The specific number and location of the cameras 102 on the vehicle 100 are determined according to the actual situation.

[0072] The hand information collected by camera 102 mainly includes the hand gestures of the five fingers and the position information of the hand relative to camera 102. If this application uses the binocular ranging principle to determine the position information of the hand, then two cameras need to be set at each position.

[0073] The processor 103 can be an in-vehicle central control unit, a central processing unit (CPU), a cloud server, etc., used to process the images captured by the camera 102, identify the gesture category and position information of the user's hand in the image, and then control the content displayed on one display screen to move to another display screen.

[0074] Memory 104 may include volatile memory, such as random-access memory (RAM); memory 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 104 may also include combinations of the above types of memory. The data stored in memory 104 includes not only images captured by camera 102, a database of gesture categories, etc., but also various instructions, applications, etc., corresponding to methods for performing multi-screen interaction.

[0075] Figure 3 This is a flowchart illustrating a multi-screen interaction method provided in an embodiment of this application. This multi-screen interaction method can be executed by a computing device, by a processing device applied within the computing device, or by... Figure 1 The system shown executes this process. The computing device can be a terminal, such as a vehicle, an in-vehicle device (e.g., a vehicle infotainment system, in-vehicle processor, in-vehicle computer), or a cloud device such as a server. The processing device can be a chip, processing circuit, processor, etc. For ease of description, this application will use the example of a processor executing this multi-screen interaction method for detailed explanation. (Reference) Figure 3 As shown, the multi-screen interaction method includes:

[0076] S301: Acquire sensor information.

[0077] The sensing information can include information acquired through sensors, such as gesture information and environmental information acquired through one or more of camera sensors and radar sensors; it can also include environmental sound information, such as user commands, acquired through sound sensors. This sensing information can be information collected by sensors, or information processed by one or more devices such as sensors and processors, for example, noise reduction processing of image information using sensors.

[0078] In the implementation process, when a user needs to project the user interface (UI) or application (APP) displayed on one screen (hereinafter referred to as the "source screen") onto another screen (hereinafter referred to as the "target screen"), specific body movements, such as gestures (which can also be understood as wake-up gestures), can be detected by the cockpit camera to trigger the multi-screen interaction function. The multi-screen interaction function can also be triggered by voice wake-up, clicking virtual buttons on the screen, etc.

[0079] In practical implementation, gestures such as wake-up gestures and operation gestures can be implemented in various ways, such as left swipe gestures, right swipe gestures, and palm hovering gestures. These can be dynamic gestures (or dynamic trend gestures) or static gestures. The processor used for gesture recognition can obtain sensor information through interface circuits to determine the gesture or action currently being performed by the user. Below, this application introduces gestures using two solutions: "five-finger gripping for display screen projection" and "two-finger gripping for application sharing." It should be understood that this application is not limited to these two solutions.

[0080] The "five-finger grasping to achieve screen projection" solution is used as an example to introduce the gesture action. As shown in Figure 4(a), when the user starts to grasp, he first spreads his five fingers and moves them close to the source screen. When grasping, the five fingers gradually come together. During the projection process, the five fingers that are together move from the source screen position to the target screen position, and then gradually move closer to the target screen and gradually spread his five fingers.

[0081] Taking the "two-finger grab to share applications" solution as an example, the gesture action is introduced as shown in Figure 4(b). When the user starts grabbing, he first spreads his five fingers (or spreads his thumb and index finger and bends the other fingers toward his palm) and then moves them close to the source display screen. When grabbing, his thumb and index finger gradually come together and the other fingers bend toward his palm. During the screen projection process, the two fingers that are together move from the source display screen to the target display screen and then gradually move closer to the target display screen, while gradually spreading his thumb and index finger.

[0082] It should be understood that the gesture for enabling multi-screen interaction in the embodiments of this application can be the above-mentioned "spread five fingers" gesture, or the gesture of "spreading five fingers and gradually moving closer to the display screen", or other gestures. For ease of explanation, the above gesture will be used as an example for further description below.

[0083] Gesture information can be determined by acquiring images (or video streams) from a camera, such as identifying a person's fingers and the gestures displayed by each finger in an image using a gesture recognition algorithm; it can also be obtained by acquiring radar information, such as extracting the features of a person's fingers in the image through a neural network after obtaining a 3D point cloud image, and then determining the gestures displayed by the fingers based on the finger features; it can also be determined by other information, which is not limited here. The following uses images or video streams captured by a camera as an example to describe the technical solution of this application.

[0084] After enabling the multi-screen interaction function, the processor acquires sensor information corresponding to the location of the source display screen. This sensor information can be data collected by sensors such as cameras and radar covering the control area of ​​the source display screen, or processed data. For example, as shown in Figure 2(a), after receiving the instruction to enable the interaction function from display screen 101-2, the processor activates the cameras to avoid activating all cameras in the vehicle, thus saving vehicle power and protecting the privacy of passengers in other seats within the vehicle 100.

[0085] After receiving sensor information, the processor performs hand target detection and gesture recognition to identify the user's hand gesture. For example, the processor receives an image including a hand captured by a camera, such as... Figure 5 As shown, the acquired image can first be preprocessed, including denoising and information enhancement. Then, a hand target detection algorithm can be used to obtain the target gesture in the image. Next, keypoint estimation is performed on the 21-degree-of-freedom (DOFS) or 26-DOFS (or more) two-dimensional (2D) coordinates of the hand to obtain the classification and description of the current gesture. Finally, a gesture recognition algorithm is used to identify the target gesture, recognizing the hand gesture displayed in the image.

[0086] In this application, the processor recognizes specific gestures such as wake-up gestures or operation gestures in a frame of an acquired image or video stream. Since the interactive function is enabled, the camera continuously captures images or video streams, and the processor continuously acquires and processes these images or video streams. Therefore, to demonstrate the temporal sequence of the acquired images or video streams, this application describes the processor numbering the images in the acquired images or video streams according to chronological order, defining the acquired image at this moment as the i-th image or the j-th frame of the video stream, where i and j are both positive integers greater than 0.

[0087] As shown in Figures 4(a) and 4(b), a specific gesture is a dynamic trend gesture. In the process of determining a specific gesture, the processor can compare the distance between the fingers in the image obtained at the current moment with the distance between the fingers in the image obtained at the previous moment. If the distance between the five fingers gradually decreases, it indicates the "five fingers together" gesture. If the distance between the five fingers gradually decreases and the middle finger, ring finger and little finger are bent, it indicates the "thumb and index finger gradually come together, and the other fingers are bent towards the palm" gesture.

[0088] When the processor detects a specific gesture, it controls the source display to enter a multi-screen interaction state, so that the source display can extract the displayed interface or the interface presented by the application based on the change of the gesture. The processor can also calculate the spatial coordinates (x0, y0, z0) of the specific gesture relative to the camera using a 3D hand positioning algorithm based on the image acquired by the camera.

[0089] As shown in Figure 2(a), if the camera is a time-of-flight (TOF) camera, a binocular camera, or other camera capable of measuring depth, the processor acquires the image captured by the camera and the depth information provided by the camera. Then, using the camera's intrinsic and extrinsic parameters, it directly calculates the spatial coordinates of the optical center of the specific gesture relative to the camera. Specifically, a TOF camera can continuously send light pulses to the target object and use a sensor to receive the light returning from the target object, obtaining the distance to the target object based on the round-trip time of the detected light pulses.

[0090] If the camera is a monocular camera that cannot measure depth, the processor can calculate the depth or distance of the gesture using depth estimation methods. Several possible depth estimation methods are given here as examples; this application does not limit these methods. For example:

[0091] (1) Based on the perspective-n-point (PNP) algorithm. Using a hand model prior, key points at the palm are detected. Combined with the camera model, the hand position is roughly estimated based on the PNP algorithm. Since only the position estimation is needed for the clenched fist gesture, the key point detection model can perform depth optimization for the clenched fist gesture. Although individual differences make the absolute accuracy of the 3D position of the key points obtained in this way not high, since the positional accuracy required for gesture direction judgment is not strict, this method can also be applied to determine the relative amount of directional movement.

[0092] (2) Monocular Depth Estimation Network Model Based on Deep Learning. After acquiring images captured by the camera, the monocular depth estimation network model based on deep learning can directly estimate the depth of the monocular camera, thereby generating a virtual 3D camera to compensate for the lack of a hardware 3D camera, and thus predict the depth information of a specific gesture in the next frame or multiple frames of images in the acquired sensor information.

[0093] Finally, based on the image acquired by the camera and the predicted depth information of that image, the processor uses the camera's intrinsic and extrinsic parameters to calculate the spatial coordinates of a specific gesture relative to the camera's optical center.

[0094] S302: Based on the gesture information, trigger the first display screen to display the first interface image. Here, the first display screen is the source display screen, the first interface image is the interface image displayed on the source display screen, and the first interface image also includes a sub-image. The sub-image can be a scaled-down version of the first interface image, or it can be an icon of an app, etc. The displayed content can be the entire screen displayed by the first interface image, or it can be a portion of the screen displayed by the first interface image.

[0095] Upon detecting a specific gesture, the processor can convert part or all of the current screen displayed on the source display into a sub-image smaller than the source display size, or convert an app icon displayed on the source display into a sub-image and display it on the source display, for example, in the center or left side of the source display. Optionally, after detecting a specific gesture, the processor can also regenerate a sub-image smaller than the source display size, and the content displayed is synchronized with the interface originally displayed on the source display.

[0096] For example, such as Figure 6 As shown, a time bar can be set at the top of the source display screen, and the middle part is the area where the video is playing. When a specific gesture is a "five-finger together" gesture, the processor will crop the entire interface currently displayed on the source display screen (including the time bar and the video area) into an image and display it in the middle of the source display screen.

[0097] like Figure 7As shown, when the specific gesture is "the thumb and index finger gradually come together, and the other fingers bend towards the palm", the processor will crop the area where the video is displayed on the source display screen at the current moment into an image and display it in the middle of the source display screen.

[0098] for Figure 7 The decision of whether to convert the time bar or the video playback area into a sub-image can be implemented in various ways. For ease of understanding, several possible implementations are presented below as examples; this application does not limit the scope of the proposed implementation. For example:

[0099] Prioritize applications, such as setting navigation apps as the first priority, video, music, and radio apps as the second priority, basic apps like time and weather forecasts as the third priority, and so on. By setting different priorities for different applications, when converting sub-images, the content of the application with the highest priority displayed on the source screen is converted into a sub-image to ensure that the content displayed in the sub-image is the content that the user wants to receive.

[0100] For example, the distance between the position of a specific gesture and various areas on the source display screen can be used to determine the sub-image. If the gesture is close to the video playback area, the video playback area is converted into a sub-image; if the gesture is close to the time bar, the time bar is converted into a sub-image. This application is not limited to the above two solutions and may also include other solutions.

[0101] The size of the sub-image displayed on the source display screen can be a fixed value or a variable value. Optionally, as the processor continuously acquires images from the camera, if the distance between fingers in a recognized specific gesture continuously decreases, the size of the sub-image displayed on the source display screen can also be continuously reduced.

[0102] In addition, the processor-controlled source display screen not only presents sub-images, but can also display the identifiers of other displays that can receive sub-images, as well as positional information such as the relative positions of other displays to itself. Since the positions of each display screen are fixed when the vehicle leaves the factory, the positional information of other displays around each display screen is fixed, and the positional information between the displays can be pre-stored in memory.

[0103] Optionally, the orientation information between the displays can be represented by a vector formed between the positions of the two displays. Taking Figure 2(a) as an example, for display 101-2, with the center of display 101-2 as the origin of the spatial coordinate system (X0, Y0, Z0), since the position of each display is fixed, the coordinate positions between each display can be stored in advance. After determining the coordinates (X, Y, Z) of each other display in the coordinate system with the center of display 101-2 as the origin of the spatial coordinate system, the vector M of each other display relative to display 101-2 is calculated, and the calculated vector M is used as the orientation of each other display relative to display 101-2.

[0104] The processor can control the display of position information for other displays on the source display, allowing the user to intuitively understand how to move the sub-image onto the target display. Figure 6 For example, and in conjunction with Figure 2(a), since the source display screen is to share the video being played to other display screens, and all other display screens on the vehicle can display the video, a virtual display screen pattern can be created on the source display screen. At the same time, the orientation and name of each display screen relative to the source display screen are displayed in text form, and the pattern corresponding to each display screen is set at the edge of the source display screen near the physical display screen according to the orientation relationship.

[0105] When the processor controls the source display screen to generate a sub-image that is displayed in the center of the source display screen, it can obtain the spatial coordinates of the gesture in the camera image acquired at the corresponding moment, and establish a mapping relationship between the position of the sub-image on the source display screen at this moment and the spatial coordinates of the gesture. This allows the sub-image to move in the corresponding direction and proportional distance relative to the current position of the sub-image when the gesture moves.

[0106] It should be noted that the processor can receive every image or video stream captured by the camera in real time, or it can receive images or video streams captured by the camera at intervals. This application does not limit this. Therefore, the phrase "the next image or the next frame" mentioned below does not mean two consecutive images or two frames captured by the camera, but can be several or dozens of images (frames) in between.

[0107] If, after detecting a specific gesture, the processor fails to detect that gesture in one or more subsequent frames, it can control the source display to stop displaying the sub-image and location information. For example, if the processor fails to recognize the specific gesture in the next frame or frames after the initial detection, it could be due to changes in the user's gesture or the gesture moving outside the camera's field of view. In such cases, the processor can send a control command to the source display to prevent it from displaying the sub-image and location information.

[0108] Optionally, to avoid accidental triggering of interactive functions, the processor determines to trigger the function corresponding to a specific gesture after detecting a specific gesture more than a certain number of times. For example, if the processor acquires the specific gesture multiple times in the sensor information within a preset time, the processor can consider it a valid gesture and trigger the corresponding function. As another example, if multiple frames in the images acquired by the camera within a preset time detect a specific gesture, exceeding a preset threshold, the processor can send a control command to the source display screen to display a sub-image and location information; if the number of images containing the specific gesture is less than the preset threshold, the processor will not perform subsequent operations and will discard or delete the detected result.

[0109] If the processor detects that the next frame or multiple frames also include a specific gesture, and the position of the specific gesture changes compared to the previous frame or the image in which the specific gesture was first detected (i.e., the i-th image or the j-th frame in the video stream), it can calculate the movement vector m = (x-x0, y-y0, z-z0) of the specific gesture based on the spatial coordinates (x0, y0, z0) of the specific gesture calculated from the next frame or multiple frames and the spatial coordinates (x, y, z) of the specific gesture calculated from the previous frame or the image in which the specific gesture was first detected. Then, the movement vector m is compared with the orientation vector M. Based on the comparison result, the display screen corresponding to the orientation vector M that is parallel to or has the smallest angle with vector m is determined as the target display screen.

[0110] S303: The trigger sub-image is displayed on the second display screen. The condition for displaying the trigger sub-image on the second display screen is: the distance between the detected first position information and the second position information is greater than a set threshold.

[0111] The first location information refers to the spatial coordinates of a specific gesture relative to the camera in the image where the feature gesture is first detected (i.e., the i-th image or the j-th frame in the video stream). The second location information refers to the spatial coordinates of a specific gesture relative to the camera in the next frame or more frames later (i.e., the (i+n)-th image or the (j+n)-th frame in the video stream, where n is a positive integer greater than zero). The specific implementation process is as follows:

[0112] For example, after the processor calculates the motion vector *m* in the image captured by the camera one or more frames after a specific gesture, it combines this with the normal vector *n* of the source display plane to calculate the translation vector *m1* projected onto the source display plane. Then, the processor can determine the direction of movement of the sub-image on the source display plane based on the direction of the translation vector *m1*, and determine the distance the sub-image moves on the source display plane based on the magnitude of the translation vector *m1* and a pre-set proportional relationship between the magnitude and the distance the sub-image moves on the source display plane. Finally, the processor can control the sub-image on the source display plane to move from the center of the plane along the direction of the translation vector *m1* by a certain proportional relationship of the magnitude of the translation vector *m1*, thereby enabling the sub-image to move in accordance with the user's gesture.

[0113] As the user moves using specific gestures, the processor can send a control command to the source display screen based on each received image processing result, causing the sub-image to appear at different positions on the source display screen, thus enabling the sub-image to move with the specific gesture. If the processor detects that part or all of the interface is not displayed on the source display screen after the sub-image moves with the specific gesture, the processor can simultaneously send a control command to the target display screen, causing the target display screen to show the interface where the sub-image is not displayed on the source display screen.

[0114] Optionally, if the sub-image is an app icon, the processor can determine when the sub-image is displayed on the target display screen based on the distance moved by a specific gesture. The controller then sends a control command to the target display screen, which is used to make the target display screen display the app icon or to make the target display screen display the interface after the app is running.

[0115] Regarding the criteria for determining how to move a sub-image from the source display screen to the target display screen, this application provides two examples of criteria, which are not limited here. For example:

[0116] 1. The magnitude of the translation vector m1. This application pre-sets a threshold. After the processor determines the target display screen based on the translation vector m, it calculates the magnitude of the translation vector m1 based on the translation vector m. If the magnitude is greater than the set threshold, the sub-image is moved to the target display screen; if the magnitude is not greater than the set threshold, the sub-image remains on the source display screen.

[0117] 2. The center point of the sub-image moves out of the source display screen. This application uses the movement of the center point of the sub-image out of the source display screen as the boundary. During the movement of the sub-image with a specific gesture, the processor can detect the position of the sub-image on the source display screen in real time. If the processor detects that the center point of the sub-image is on the border of the source display screen or not on the source display screen, it indicates that half of the area of ​​the sub-image has moved out of the source display screen, and then the sub-image is moved to the target display screen; if the processor detects that the center point of the sub-image is on the source display screen, it indicates that half of the area of ​​the sub-image has not moved out of the source display screen, and then the sub-image continues to move on the source display screen.

[0118] Of course, this application may also use the area of ​​the sub-image removed, the size of the long side of the sub-image, etc. as judgment conditions, which are not limited here.

[0119] When a sub-image moves on the source display screen but its center point does not move off the source display screen, the effect displayed on the source display screen is as follows: Figure 8 As shown. The target display screen may or may not display the portion of the sub-image that has been moved out of the source display screen, or it may display the portion of the sub-image that has been moved out of the source display screen, as shown. Figure 9 As shown, this allows users to more intuitively see which display screen they are moving the sub-image to and the effect of the movement.

[0120] If the resolution of the target display is different from that of the source display, after the sub-image is moved to the target display, the processor can increase or decrease the resolution of the sub-image to match the resolution of the target display before sending it to the target display for display, so as to avoid abnormal display of the sub-image when it is moved to the target display.

[0121] If the size or aspect ratio of the sub-image is different from that of the target display screen, or if the size or aspect ratio of the source display screen is different from that of the target display screen, the processor, when determining whether to move the sub-image to the target display screen, can compare the size or aspect ratio of the two and adjust the size or aspect ratio of the sub-image so that the sub-image can be displayed correctly on the target display screen. For example, if the size, long side, and short side of the target display screen are all larger than the source display screen, the processor can allow the sub-image to maintain its original size on the target interface. Figure 10 As shown, the size of the sub-image can also be increased so that the sub-image is displayed at its maximum size on the target display screen, such as... Figure 11As shown; if the size of the target display screen, its long side, and its short side are all smaller than the size of the source display screen or the sub-image, the processor can reduce the size of the sub-image so that it can be displayed correctly on the target display screen; if either the long side or the short side of the target display screen is smaller than the source display screen or the sub-image, the processor can reduce the size of the sub-image until its long side matches the long side of the target display screen, or reduce the size of the sub-image until its short side matches the short side of the target display screen, and then display it on the target display screen, as shown. Figure 12 As shown.

[0122] After the sub-image moves to the target display screen, the processor should receive a camera image that does not include a specific gesture. At this point, for the user, the gesture should change from the specific gesture to a non-specific gesture such as "spreading five fingers" or "spreading the thumb and forefinger." For example, if the processor does not receive a camera image containing a specific gesture within a set time period, or detects a specific gesture such as "spreading five fingers and gradually moving closer to the target display screen," or receives three consecutive taps on the target display screen, the "multi-screen interaction" function can be disabled to save power and computational costs.

[0123] After the sub-image moves to the target display screen, if the processor receives a camera image captured by the camera that also includes a specific gesture, it can be assumed that the user is sharing the content displayed on the source display screen for the second time. At this time, the processor can execute the process in steps S301 to S303 again.

[0124] During the execution of steps S301 to S303, if the processor detects two or more specific gestures in the image captured by the camera, it can be assumed that the user is sharing the content displayed on the source display screen to two or more target display screens. The processor can first mark the two specific gestures, and then, based on the marked multiple specific gestures, execute the processes implemented in steps S301 to S303, thereby achieving multi-screen sharing of the content displayed on the source display screen to multiple target display screens simultaneously. For example, as shown... Figure 13 As shown, when the processor can share the content displayed on the source display screen with two target displays, the source display screen can simultaneously display information such as the sub-images corresponding to the two specific gestures and the direction of movement of the sub-images, so that users can intuitively see the process of the movement of the two sub-images.

[0125] In this embodiment, after the processor detects that the image captured by the camera includes a specific gesture, it can convert the content displayed on the source display screen into a sub-image and display the identifiers of other displays that can display the sub-image and their orientation information relative to the source display screen on the source display screen. The user can intuitively see the direction of movement of the subsequent gesture. Then, based on the direction of movement of the specific gesture, the target display screen is determined, and the sub-image is controlled to move on the source display screen with the movement of the specific gesture. When the distance of movement of the specific gesture is detected to be greater than a set threshold, the sub-image is moved to the target display screen, thereby realizing the multi-screen interaction function.

[0126] Figure 14 This is a schematic diagram illustrating the gesture and sub-image movement process of using five fingers to achieve screen projection functionality, as provided in an embodiment of this application. Figure 14 As shown, the driver's gesture changes and movements during the process of pushing content from the source display to the target display are as follows:

[0127] First, place one hand in front of the source display screen, with your five fingers spread out, such as... Figure 14 As shown in (a); then, the five fingers gradually come together (and move away from the source display), activating the source display to start the screen-flying mode, and converting the content currently displayed on the interface into a sub-image, such as... Figure 14 As shown in (b); then, the joined fingers move to the left (or towards the target display screen), while the finger gesture remains unchanged. The sub-image on the source display screen moves with the finger's movement in the same direction, as shown in (b). Figure 14 As shown in (c); finally, when the distance the sub-image has moved meets the set conditions, it will be automatically displayed on the target display screen, as shown in (c). Figure 14 (d) The gesture at this point can still be as shown. Figure 14 The state shown in (c) can also be the five fingers that are joined together gradually opening (and moving towards the target display screen).

[0128] The screen mirroring function is implemented as follows:

[0129] Figure 14 The state shown in (a) represents the execution state for the processor. Figure 3 In step S301 of the flowchart shown, the image captured by the camera is acquired and recognized; for the source display screen, no sub-image, other display screen identifiers, or orientation information are displayed at this time; for the target display screen, no sub-image is displayed at this time; for the user, the user is spreading their five fingers and approaching the source display screen.

[0130] Figure 14 The state shown in (b) indicates that, for the processor, execution... Figure 3In step S302 of the flowchart shown, after detecting that the image captured by the camera includes a specific gesture, the source display screen is controlled to display information such as a sub-image, other display screen icons, and orientation information. For the source display screen, the sub-image shrunk to the center of the interface, other display screen icons, and orientation information are displayed at this time. For the target display screen, no sub-image is displayed at this time. For the user, the user's gesture is changing from an open five-finger gesture to a gesture that gradually closes.

[0131] Figure 14 The state shown in (c) represents the processor's execution. Figure 3 In step S302 of the flowchart shown, after a specific gesture movement is detected, the movement of the sub-images on the target display screen and the control source display screen is determined according to the movement direction and distance. For the source display screen, the sub-image is displayed as it moves. For the target display screen, no sub-image is displayed (or only a portion of the sub-image is displayed). For the user, the user's gesture moves from the source display screen to the target display screen.

[0132] Figure 14 The state shown in (d) represents the execution state for the processor. Figure 3 In step S303 of the flowchart shown, after detecting that the distance of a specific gesture movement is greater than a set threshold, the sub-image is controlled to move to the target display screen. For the source display screen, the sub-image, other display screen logos, and orientation information are not displayed at this time. For the target display screen, the sub-image is displayed at this time. For the user, the user's gesture moves out of the camera's shooting area (or the user spreads their five fingers apart).

[0133] From the processor's internal execution perspective, after enabling multi-screen interaction, the processor continuously recognizes the images captured by the camera. First, a hand target detector is run. Then, the hands detected by the target detector are further classified to determine if a screen-flying trigger gesture exists (i.e., the specific gesture mentioned above, the "five fingers spread" gesture). If a trigger gesture exists, a target tracking algorithm is run to track the position and state of the trigger hand, and a prompt to activate screen-flying is displayed on the source screen. If the tracked hand switches to an activation gesture (i.e., the specific gesture mentioned above, "five fingers together" or "two fingers together"), then screen-flying is activated according to the activation gesture, and the process enters... Figure 3In step S302, the target direction for screen flying is displayed on the source display screen. If the activated hand moves at this time, the movement vector (x-x0, y-y0, z-z0) is calculated based on the current hand coordinates (x, y, z). Then, the translation vector m1 of the screen projection plane is calculated based on the normal vector n of the screen plane. The source display screen displays the movement effect based on the distance and magnitude of the vector m1. If the movement distance, i.e., the magnitude of m1, is greater than a threshold, the screen flying action is performed on the target display screen. If it is less than the threshold, the activation gesture is abandoned, and the current screen flying operation can be canceled.

[0134] Figure 15 This is a schematic diagram of a multi-screen interaction device provided in an embodiment of this application. The multi-screen interaction device can be a computing device or electronic device (e.g., a terminal), or a device within an electronic device (e.g., an ISP or SoC). It can achieve, for example... Figures 3 to 14 The multi-screen interaction method and the above-described optional embodiments are shown. Figure 15 As shown, the multi-screen interactive device 1500 includes a transceiver unit 1501 and a processing unit 1502.

[0135] In this application, the multi-screen interaction device 1500 is specifically implemented as follows: the transceiver unit 1501 is used to acquire sensing information, which includes gesture information; the processing unit 1502 is used to trigger the first display screen to display a first interface image based on the gesture information, the first interface image including a sub-image, the movement trend of the sub-image being associated with the gesture information; and trigger the sub-image to be displayed on the second display screen.

[0136] The transceiver unit 1501 is used to execute S301 and any optional example thereof in the above-described multi-screen interaction method. The processing unit 1502 is used to execute S302, S303 and any optional example thereof in the above-described multi-screen interaction method. For details, please refer to the detailed description in the method examples, which will not be repeated here.

[0137] It should be understood that the multi-screen interaction device in this application embodiment can be implemented by software, for example, by a computer program or instructions with the above-mentioned functions. The corresponding computer program or instructions can be stored in the memory inside the terminal, and the above functions can be implemented by the processor reading the corresponding computer program or instructions in the memory. Alternatively, the multi-screen interaction device in this application embodiment can also be implemented by hardware. The processing unit 1502 is a processor (such as an NPU, GPU, or processor in a system chip), and the transceiver unit 1501 is a transceiver circuit or interface circuit. Alternatively, the multi-screen interaction device in this application embodiment can also be implemented by a combination of a processor and a software module.

[0138] It should be understood that the processing details of the apparatus in the embodiments of this application can be referred to Figure 4- Figure 14The relevant content illustrated in this application will not be repeated in the embodiments.

[0139] Figure 16 This is a schematic diagram of another multi-screen interaction device provided in an embodiment of this application. This multi-screen interaction device can be a computing device or electronic device (e.g., a terminal), or a device within an electronic device (e.g., an ISP or SoC). It can achieve, for example... Figures 3 to 14 The multi-screen interaction method and the above-described optional embodiments are shown. Figure 16 As shown, the multi-screen interaction device 1600 includes: a processor 1601 and an interface circuit 1602 coupled to the processor 1601. It should be understood that, although... Figure 16 Only one processor and one interface circuit are shown in the diagram. The multi-screen interaction device 1600 may include other numbers of processors and interface circuits.

[0140] In this application, the multi-screen interaction device 1600 is specifically implemented as follows: the interface circuit 1602 is used to acquire sensing information, which includes gesture information; the processor 1601 is used to trigger the first display screen to display a first interface image based on the gesture information, the first interface image including a sub-image, the movement trend of the sub-image being associated with the gesture information; and to trigger the sub-image to be displayed on the second display screen.

[0141] The interface circuit 1602 is used to communicate with other components of the terminal, such as memory or other processors. The processor 1601 is used to interact with other components via the interface circuit 1602. The interface circuit 1602 can be an input / output interface for the processor 1601.

[0142] For example, processor 1601 reads computer programs or instructions from a coupled memory via interface circuit 1602, and decodes and executes these computer programs or instructions. It should be understood that these computer programs or instructions may include the aforementioned terminal function programs, or the function programs of the multi-screen interaction device applied within the terminal. When the corresponding function program is decoded and executed by processor 1601, the terminal or the multi-screen interaction device within the terminal can implement the scheme in the multi-screen interaction method provided in the embodiments of this application.

[0143] Optionally, these terminal function programs are stored in external memory of the multi-screen interaction device 1600. When the terminal function program is decoded and executed by the processor 1601, part or all of the contents of the terminal function program are temporarily stored in the memory.

[0144] Optionally, these terminal function programs are stored in the memory inside the multi-screen interaction device 1600. When the terminal function programs are stored in the memory inside the multi-screen interaction device 1600, the multi-screen interaction device 1600 can be installed in the terminal of this application embodiment.

[0145] Optionally, some of the content of these terminal function programs is stored in external memory of the multi-screen interaction device 1600, while other parts of these terminal function programs are stored in internal memory of the multi-screen interaction device 1600.

[0146] It should be understood that Figures 15 to 16 Any of the multi-screen interactive devices shown can be combined with each other. Figure 1 To Figure 2, Figure 4 Figure 12 The design details of any of the multi-screen interaction devices and their respective optional embodiments can be referenced interchangeably or further. Figures 10 to 12 The multi-screen interaction method shown in the illustration, along with the design details of each optional embodiment, will not be repeated here.

[0147] It should be understood that Figure 3 The multi-screen interaction method and its various optional embodiments are shown. Figures 15 to 16 The multi-screen interaction device and its various optional embodiments shown can be used not only to process video or images during shooting, but also to process video or images that have already been shot. This application does not limit the scope of the application.

[0148] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described above.

[0149] This application provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements any of the methods described above.

[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0151] Furthermore, various aspects or features of the embodiments of this application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). Additionally, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0152] In the above embodiments, Figure 15 The multi-screen interactive device 1500 can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0153] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0154] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for multi-screen interaction, comprising: Acquire sensor information, including gesture information; Based on the gesture information, a first display screen is triggered to display a first interface image. The first interface image includes a sub-image and position information. The movement trend of the sub-image is associated with the gesture information. The position information is an identifier of at least one other display screen that can display the sub-image. The location information is displayed at the edge of the first display screen according to the orientation information of the at least one display screen relative to the first display screen; The sub-image is triggered to be displayed on the second display screen; The sensing information includes sensing information acquired by at least one of a camera, radar, and sound sensor.

2. The method of claim 1, wherein the position information is at least one display screen identification on which the sub-image can be displayed, and wherein the method further comprises: determining the at least one display screen identification on which the sub-image can be displayed based on the position information. The display screen identifier includes text information, which indicates the second display screen.

3. The method according to claim 1 or 2, wherein the sensing information is obtained, characterized in that, Based on the location of the first display screen, obtain the corresponding location's sensor information.

4. The method according to claim 3, wherein the first interface image includes a sub-image and location information, characterized in that, The sub-image includes a single image captured from the entire interface displayed on the first display screen at the current moment.

5. The method according to claim 4, characterized in that, The sub-image is displayed in the center of the first display screen.

6. The method according to claim 1, characterized in that, The sensing information includes multiple gesture information within a preset time period.

7. A multi-screen interactive device, comprising: A transceiver unit is used to acquire sensor information, including gesture information. The transceiver unit includes at least one of a camera, radar, and sound sensor. The processing unit is configured to trigger a first display screen to display a first interface image based on the gesture information. The first interface image includes a sub-image and position information, wherein the movement trend of the sub-image is associated with the gesture information; and the position information is an identifier of at least one other display screen that can display the sub-image. The location information is displayed at the edge of the first display screen according to the orientation information of the at least one display screen relative to the first display screen; The sub-image is triggered to be displayed on the second display screen.

8. The apparatus according to claim 7, characterized in that, The display screen identifier includes text information, which indicates the second display screen.

9. The apparatus according to claim 7 or 8, characterized in that, The location information is displayed at the edge of the first display screen according to the orientation information of the at least one display screen relative to the first display screen.

Images