Control method and device of expansion screen assembly, computer device and storage medium

Abstract

The application relates to the technical field of intelligent wearing, and specifically discloses a control method and device of an extended screen assembly, computer equipment and a storage medium. The screen contour of a user equipment is recognized to activate a screen extension mode; based on the relative position relationship between each extended screen assembly and the screen contour, the extended screen assembly and the screen of the user equipment are displayed in a display interface; the coordinates of a first key point and the coordinates of a second key point of a user's hand are tracked to generate a moving selection pointer; when the user's hand moves, the extended screen assembly intersecting with the moving selection pointer is determined as a target assembly to control the target assembly. The method obtains the moving selection pointer through the two key points of the user's hand, and then obtains the intersection condition of the moving selection pointer and the extended screen assembly when the user moves the hand, and the extended screen assembly intersecting with the moving selection pointer is taken as the target assembly, so that the user operation is simplified, and the portability and accuracy of the user in selecting the extended screen assembly are improved.

Description

Technical Field

[0001] This application relates to the field of smart wearable technology, and in particular to a control method, device, computer equipment, and storage medium for an extended screen component. Background Technology

[0002] Currently, although some smart glasses (such as AR (Augmented Reality) glasses) and extended screen products for user devices (such as mobile phones and tablets) are available on the market, most of them rely on gestures or voice commands to operate. This requires users to spend time learning and adapting to these methods. Furthermore, if user gestures are unclear or verbal expressions are ambiguous, the selected extended screen may not be what the user needs, resulting in low accuracy in screen selection. Therefore, improving the portability and accuracy of extended screen selection for users has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a control method, apparatus, computer device, and storage medium for an extended screen component to improve operational portability and the accuracy of extended screen selection.

[0004] In a first aspect, this application provides a method for controlling an extended screen component, the method comprising:

[0005] Identify the screen outline of the user device to activate the screen extension mode;

[0006] Based on the relative positional relationship between each extended screen component and the screen outline, the extended screen components and the screen of the user device are displayed in the display interface;

[0007] Based on a preset algorithm, the coordinates of the first key point and the second key point of the user's hand in the display interface are tracked to generate a movement selection pointer.

[0008] When the user's hand movement is detected, the extended screen component that intersects with the movement selection pointer is identified as the target component, and the target component is controlled.

[0009] Secondly, this application also provides a control device for an extended screen assembly, the device comprising:

[0010] The screen outline recognition module is used to identify the screen outline of the user device in order to activate the screen extension mode.

[0011] The display interface module is used to display the extended screen components and the screen of the user device in the display interface based on the relative positional relationship between each extended screen component and the screen outline.

[0012] The moving selection pointer generation module is used to generate a moving selection pointer by tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm.

[0013] The target component determination module is used to determine the extended screen component that intersects with the movement selection pointer as the target component when the user's hand is detected to move, so as to control the target component.

[0014] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the control method of the extended screen component as described above.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the control method for the extended screen component as described above.

[0016] This application discloses a control method, apparatus, computer device, and storage medium for extended screen components. The method identifies the screen outline of a user device to activate a screen extension mode. Based on the relative positional relationship between each extended screen component and the screen outline, the extended screen components and the user device screen are displayed on a display interface. Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a movement selection pointer. When movement of the user's hand is detected, the extended screen component intersecting with the movement selection pointer is identified as the target component, and the target component is controlled. This method obtains the movement selection pointer through two key points of the user's hand, and then obtains the intersection of the movement selection pointer and the extended screen components when the user moves their hand. The extended screen component intersecting with the movement selection pointer is used as the target component, simplifying user operation and improving the portability and accuracy of selecting extended screen components. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a first embodiment of a control method for an extended screen component provided in this application;

[0019] Figure 2This is a schematic diagram of the display interface of a control method for an extended screen component provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the adaptive change effect of a control method for an extended screen component provided in an embodiment of this application.

[0021] Figure 4 This is a schematic flowchart of a second embodiment of a control method for an extended screen component provided in this application;

[0022] Figure 5 This is a schematic flowchart of a third embodiment of a control method for an extended screen component provided in this application;

[0023] Figure 6 A schematic block diagram of a control device for an extended screen assembly provided for an embodiment of this application;

[0024] Figure 7 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0027] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] Embodiments of this application provide a control method, apparatus, computer device, and storage medium for an extended screen component. The control method for the extended screen component can be applied to a server, generating a movement selection pointer based on hand key points to determine the target component, thus improving the portability and accuracy of user selection of the extended screen component. The server can be a standalone server or a server cluster.

[0030] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a first embodiment of a control method for an extended screen component provided in this application. This control method for the extended screen component can be applied in a server to generate a movement selection pointer based on hand key points to determine the target component, thereby improving the portability and accuracy of user selection of the extended screen component.

[0032] like Figure 1 As shown, the control method for the extended screen component specifically includes steps S101 to S1034.

[0033] S101. Identify the screen outline of the user device to activate the screen extension mode.

[0034] In one embodiment, the screen brightness changes when the user turns on the screen of a user device (such as a mobile phone, tablet, etc.).

[0035] In a specific embodiment, when the smart wearable device (such as AR glasses, AR helmets, etc.) detects that the user device's screen is lit, it invokes its own image acquisition module to identify the screen outline of the user device and activate the screen extension mode. It is understood that the image acquisition module can be a camera built into the smart wearable device; in another embodiment, the image acquisition module can also be a camera connected to the smart wearable device.

[0036] In one embodiment, such as Figure 2 As shown, in screen extension mode, the user device and the extension screen component are bound together. The display interface shown to the user by the smart wearable device will show the screen of the user device and the extension screen component of the user device or the smart wearable device.

[0037] Understandably, extended screen components can be application components on a user device that can display the application content of the user device; for example, a video component can display a playing video.

[0038] In another embodiment, the extended screen component can also be a functional component in a smart wearable device, which can display the relevant functions of the smart wearable device so that users can query and operate it.

[0039] Among them, the extended screen component is a high-performance cross-terminal development framework for smart large-screen devices, and a fast application engine that runs on smart wearable devices. It can help developers quickly build application services for smart wearable devices and provide multi-screen interaction capabilities.

[0040] S102. Based on the relative positional relationship between each extended screen component and the screen outline, display the extended screen components and the screen of the user device in the display interface.

[0041] Based on the relative positional relationship between each extended screen component and the screen outline, before displaying the extended screen components and the screen of the user device in the display interface, the method further includes: obtaining the outline coordinates of the screen outline; obtaining the first position of the user device in the display interface based on the outline coordinates; and obtaining the relative positional relationship between each extended screen component and the user device based on the preset size of each extended screen component, the screen size of the user device, and the first position.

[0042] In one embodiment, the smart wearable device invokes an image acquisition module to identify the screen outline of the user device and records its coordinates. From these coordinates, the top-left and bottom-right corner coordinates of the screen are selected to represent the screen's initial position on the display interface. For example, after a user turns on their phone screen, the AR glasses identify the screen's outline and obtain the coordinates of its top-left and bottom-right corners to represent the screen's initial position.

[0043] In one embodiment, a smart wearable device (such as smart glasses) projects the screen content of the user device or related content of the smart wearable device itself as AR content into the external space of the screen outline in the form of an extended screen component, forming an extended display of the user device screen.

[0044] In one embodiment, the display interface is the interface shown to the user by the smart wearable device, and is captured by the image acquisition module of the smart wearable device. It is understood that, without cropping the image acquired by the image acquisition module, the size of the image acquired by the image acquisition module is the same as the size of the display interface. The image size can be calculated based on the preset image resolution and pixel values ​​of the image acquisition device. For example, if an image has 600×800 pixels and a resolution of 300×400, then its actual length and width are: 600 / 300 = 2 inches, 800 / 400 = 2 inches, and approximately 5 centimeters in both length and width, meaning it is actually an image 5 centimeters long and 5 centimeters wide. The coordinate information of the display interface can be obtained from the image size information acquired by the image acquisition module, and the coordinate system can be determined based on this information. For example, when the image acquisition module is a camera device, and the width of the image captured by the camera is known to be c... w The height is c h Taking the top-left corner as the origin (0, 0), the leftmost edge of the display screen as the ordinate axis, and the top edge of the display screen as the abscissa axis, we can obtain the coordinates of the bottom-right corner of the display screen as (v... w c h ).

[0045] In one embodiment, the relative positional relationship between the extended screen components and the user device's screen size is determined based on the dimensions of each extended screen component and the user device's screen size. For example, if the user device is a mobile phone with a screen size of 7.5 cm wide and 13.5 cm long, and the extended screen components have the following dimensions: first extended screen component: 2 cm × 2 cm, second component: 1.5 cm × 1.5 cm, third component: 1.5 cm × 2 cm, etc., then appropriately sized extended screen components can be arranged in rows at the top of the phone and columns on the sides of the phone according to the width and length of the phone screen to obtain the relative positional relationship between each extended screen component and the phone screen. For example, the first extended screen component is located in the first column and second row on the left side of the phone screen.

[0046] In one embodiment, the position coordinates of each extended screen component are determined based on its relative position to the user device's first position and the size of the extended screen component. For example, if the user device's top-left corner is (20, 30) and its bottom-right corner is (30, 50), then the first position is represented as ((20, 30), (30, 50)). If the first extended screen component is 2 cm wide and 2 cm long, and its relative position to the user device is in the top row and first column of the user device, then the coordinates of the top-left corner of the first extended screen component are (20, 28), and the bottom-right corner is (22, 32). The position coordinates of the first extended screen component can then be expressed as...

[0047] T1 (20,28)，(22,32)

[0048] The screen size of the user device is known and can be obtained through measurement or by querying the product information of the user device. The size of each extended screen component can be set by the user or preset by the developers.

[0049] The positions of each extended screen component are indicated as follows:

[0050]

[0051] Where x1y1 represents the coordinates of the top-left corner of each extended screen component, x2y2 represents the coordinates of the bottom-right corner of each extended screen component, and T1, T2, ..., Tn are used to represent the position of the 1st, 2nd, ..., nth extended screen components. In other words, the position of an extended screen component is represented by the coordinates of its top-left and bottom-right corners.

[0052] In one embodiment, based on the obtained positions of each extended screen component in the display interface and the first position of the user device's screen outline in the display interface, each extended screen component is displayed around the user device screen, for example, as... Figure 2 As shown, the various extended screen components are distributed on the left, right, and top sides of the user device screen.

[0053] S103. Based on a preset algorithm, track the coordinates of the first key point and the second key point of the user's hand in the display interface, and generate a movement selection pointer.

[0054] Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a movement selection pointer. This includes: identifying the user's hand based on the preset algorithm to determine the first key point and the second key point; when the user's hand is detected to be moving, tracking the coordinates of the first key point of the first key point and the second key point of the second key point in the display interface; and obtaining a line segment passing through the coordinates of the first key point and the second key point as the movement selection pointer, wherein the direction of the movement selection pointer is from the first key point to the second key point.

[0055] In one embodiment, the preset algorithm is a hand keypoint algorithm, which can track the coordinates of hand keypoints in real time based on the user's hand movements. For example, a hand keypoint detection method based on MediaPipe (MediaPipe is a framework for building cross-platform machine learning solutions) includes a palm detector and a hand coordinate model. The palm detector is used to determine the position of the user's hand, and the hand coordinate model is used to obtain the coordinates of the hand keypoints. When a hand is detected, the coordinates of the hand keypoints can be tracked in real time as the hand moves.

[0056] In a specific embodiment, a preset algorithm is stored in the smart wearable device. When in screen extension mode, the smart wearable device calls the preset algorithm to identify 21 key points on the hand and determines the first key point and the second key point among the key points. For example, the first joint of the user's thumb is taken as the first key point and the fingertip of the user's thumb is taken as the second key point.

[0057] In one embodiment, movement of the user's hand causes changes in the coordinates of a first keypoint and a second keypoint. Upon detecting these changes, it can be determined that the user's hand is moving.

[0058] In one embodiment, when the user's hand moves, a preset algorithm tracks the coordinates of the first key point corresponding to the first key point and the coordinates of the second key point corresponding to the second key point in real time, and uses the line segment passing through the coordinates of the first key point and the second key point as the movement selection pointer.

[0059] In one embodiment, the direction of the movement selection pointer is from the first key point to the second key point. For example, a line segment connecting the first joint of the thumb to the tip of the thumb is the movement selection pointer, and the direction from the first joint of the thumb to the tip is the direction of the movement selection pointer.

[0060] S104. When the user's hand is detected to be moving, the extended screen component that intersects with the movement selection pointer is identified as the target component, so as to control the target component.

[0061] After detecting that the user's hand is moving, and identifying the extended screen component that intersects with the movement selection pointer as the target component, the method further includes: recognizing the user's hand operation, and displaying the corresponding interface of the target component to the user based on the operation.

[0062] Based on the operation action, the corresponding interface of the target component is displayed to the user, including: recognizing the operation action, comparing the operation action with gesture actions in a preset gesture library, and determining the operation instruction corresponding to the operation action; when the operation instruction is to display a details page, displaying the details page of the target component; when the operation instruction is to switch components, switching the target component based on a preset component switching order; and when the operation instruction is to return to the display interface, exiting the details page and returning to the display interface.

[0063] In one embodiment, as the user's hand moves, the movement selection pointer changes accordingly. For example, if the user changes the direction of their thumb, the direction of the movement selection pointer generated by the user's thumb changes accordingly. During the movement, the movement selection pointer intersects with the extended screen components surrounding the user device screen in the display interface, and the extended screen component that intersects with the movement selection pointer is taken as the target component.

[0064] In one embodiment, after identifying the target component, a corresponding display instruction is generated according to the preset component selection display effect, and the target component undergoes adaptive changes, such as changes in component size or opacity. Figure 3 As shown, after selecting the target component, the target component will enlarge, achieving a floating effect, while the opacity of other unselected components will increase, highlighting the target component.

[0065] In one embodiment, after identifying the target component, the user's hand gestures are recognized, and these gestures are compared with gestures in a preset gesture library. The gesture library includes gestures and their corresponding operation instructions. After determining the gesture corresponding to the gesture, the corresponding operation instruction can be obtained. The operation instruction includes displaying a details page, switching components, returning to the main interface (i.e., the display interface), etc. The component switching can be performed according to a preset component switching order.

[0066] The above embodiments provide a control method, apparatus, computer device, and storage medium for extended screen components. The method identifies the screen outline of a user device to activate a screen extension mode; based on the relative positional relationship between each extended screen component and the screen outline, it displays the extended screen components and the screen of the user device in a display interface; based on a preset algorithm, it tracks the coordinates of a first key point and a second key point of the user's hand in the display interface to generate a movement selection pointer; when movement of the user's hand is detected, the extended screen component intersecting with the movement selection pointer is identified as the target component, and the target component is controlled. This method obtains the movement selection pointer through two key points of the user's hand, and then obtains the intersection of the movement selection pointer and the extended screen component when the user moves their hand. The extended screen component intersecting with the movement selection pointer is used as the target component, simplifying user operation and improving the portability and accuracy of user selection of extended screen components.

[0067] Please see Figure 4 , Figure 4 This is a schematic flowchart illustrating a second embodiment of a control method for an extended screen component provided in this application. This control method for the extended screen component can be applied in a server to generate a movement selection pointer based on hand key points to determine the target component, thereby improving the portability and accuracy of user selection of the extended screen component.

[0068] like Figure 4 As shown, step S104 of the control method for the extended screen component specifically includes steps S201 to S203.

[0069] S201. When the user's hand is detected to be moving, the target area is determined in the display interface based on the comparison result between the current slope of the movement selection pointer and the preset slope.

[0070] S202, The extended screen component within the target area is taken as the component to be judged;

[0071] S203. When the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, it is determined that the component to be judged intersects with the moving selection pointer, and the component to be judged is taken as the target component.

[0072] Based on the comparison result between the current slope and the preset slope of the moving selection pointer, before determining the target area in the display interface, the method further includes: obtaining the current slope of the moving selection pointer based on the coordinates of the first key point and the second key point; obtaining the coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module; determining the upper left corner and the lower right corner of the display interface based on the coordinate information, and obtaining the slope of the line segment passing through the upper left corner and the lower right corner as the preset slope.

[0073] In a specific embodiment, the display interface is the interface shown to the user by the smart wearable device, and it is captured by the image acquisition module of the smart wearable device. It is understood that, without cropping the image acquired by the image acquisition module, the size of the image acquired by the image acquisition module is the same as the size of the display interface. The size information of the display interface can be obtained from the image size information acquired by the image acquisition module, and the coordinate system can be determined based on the image size information acquired by the image acquisition module. For example, when the image acquisition module is a camera device, the width of the image acquired by the camera is known to be c. w The height is c h Taking the top-left corner as the origin (0, 0), the leftmost edge of the display screen as the ordinate axis, and the top edge of the display screen as the abscissa axis, the coordinates of the bottom-right corner of the display screen can be obtained as (c w c h The slope of the line segment formed by the top left corner and the bottom right corner is c. k , will c k As a preset slope.

[0074] In a specific embodiment, the coordinates of the first key point are set as t1, and the coordinates of the second key point are set as t2. Based on the two coordinates t1 and t2, the current slope t of the moving selection pointer can be obtained. k .

[0075] In one embodiment, based on the current slope t of the moving selection pointer k With the preset slope c k By comparing the components, the location of the target component can be preliminarily determined, and this area is designated as the target area. All extended screen components within the target area are then considered as components to be judged, awaiting determination of whether they intersect with the extension line of the moving selection pointer.

[0076] In a specific embodiment, if t k <c kThen, extend line segment t2t1 to the leftmost point x=0 on the display interface, obtaining the intersection point x0 with the leftmost edge of the display interface, thus obtaining the line segment passing through t2, t1, and x0. That is, the extension line of the moving selection pointer will pass through the left side area of ​​the user device and intersect with the preset edge of a certain extended screen component within the left side area of ​​the user device; at this point, the left side area is taken as the target area. For example... Figure 3 As shown in the first image, at this point, the current slope t of the moving selection pointer... k Less than the slope of the display interface (i.e., the preset slope) c k If the extension line of the moving selection pointer intersects with one of the extended screen components in the left area of ​​the user device screen, then the left area is taken as the target area, and all extended screen components in the left area are taken as components to be judged.

[0077] In one embodiment, if t k >c k Then, extend line segment t2t1 to the top edge of the display interface at y=0, obtaining the intersection point y0 with the top edge of the display interface, thus obtaining the line segment passing through t2, t1, and y0. That is, the extension line of the moving selection pointer will pass through the top area of ​​the user device and intersect with a preset edge of a certain extended screen component within the top area of ​​the user device. At this point, the top area is taken as the target area. For example... Figure 3 As shown in the third figure, at this point, the current slope t of the moving selection pointer... k The slope of the display interface is greater than the preset slope (c). k If the extension of the moving selection pointer intersects with one of the extended screen components in the top area of ​​the user device screen, then the top area is taken as the target area, and all extended screen components in the top area are taken as components to be judged.

[0078] In one embodiment, if t k =c k Then, extend line segment t2t1 to the origin (0,0) at the top left corner, obtaining the intersection point t0 with the top left corner of the display interface, thus obtaining the line segment passing through t2, t1, and t0. That is, the extended line of the moving selection pointer will pass through the top left corner area of ​​the user device and intersect with a preset edge of a certain extended screen component within the top left corner area of ​​the user device. At this point, the top left corner area is taken as the target area. For example... Figure 3 As shown in the second figure, at this point, the current slope t of the moving selection pointer... k Equal to the slope of the displayed interface (i.e., the preset slope) c k If the extension line of the moving selection pointer intersects with the extended screen component in the upper left corner of the user device screen, then the upper left corner area is taken as the target area, and the extended screen component in the upper left corner area is taken as the component to be judged.

[0079] In one embodiment, when a user operates with their left hand, with their finger pointing towards the right side of the user device, the extension of the moving selection pointer may intersect with a preset edge of the extended screen component within the right side area.

[0080] In one embodiment, the side of the component to be judged closest to the user device screen is used as a preset edge. For example, when the left side of the user device is determined to be the target area, the extended screen component in the left area is used as the component to be judged, and the right border of the component to be judged is used as the preset edge; when the right side of the user device is determined to be the target area, the extended screen component in the right area is used as the component to be judged, and the left border of the component to be judged is used as the preset edge; when the top area of ​​the user device is determined to be the target area, the extended screen component in the top area is used as the component to be judged, and the bottom border of the component to be judged is used as the preset edge.

[0081] When the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, it is determined that the component to be judged intersects with the moving selection pointer, and the component to be judged is taken as the target component.

[0082] Understandably, when determining the target component, the relationship between the slope of the moving selection pointer and a preset slope is first used to determine the target area where the target component might be located. All extended screen components within the target area are then considered as components awaiting determination to be the target component. Finally, using the edge of the component awaiting determination adjacent to the user device screen as a preset edge, it is determined whether the extension line of the moving selection pointer intersects with the preset edge, thus identifying a unique target component. For example, as... Figure 3 As shown in the second image, when the slope of the moving selection pointer is equal to the preset slope, the upper left corner of the user device screen is taken as the target area, and the component in the upper left corner is taken as the component to be judged. The target component is then determined from among the components to be judged. Even if the moving selection pointer still intersects with the extended screen component in the left area, since the left area is not the target area, the extended screen component in the left area has been excluded. Therefore, the moving selection pointer only intersects with the extended screen component in the upper left corner.

[0083] The above embodiments provide a control method, apparatus, computer device, and storage medium for an extended screen component. When a user's hand movement is detected, a target area is determined in the display interface based on a comparison between the current slope and a preset slope of the movement selection pointer. The extended screen component within the target area is designated as a component to be judged. When the extension line of the movement selection pointer intersects a preset edge of the component to be judged, the component to be judged is determined to intersect the movement selection pointer, and the component to be judged is designated as the target component. This method, by comparing the current slope and the preset slope of the movement selection pointer to determine the target slope, obtaining the extended screen component within the target area as the component to be judged, and determining whether the preset edge of the component to be judged intersects the extension line corresponding to the movement selection pointer, can accurately determine the extended screen component intersecting the movement selection pointer as the target component, thus improving the accuracy of target component determination.

[0084] Please see Figure 5 , Figure 5 This is a schematic flowchart illustrating a third embodiment of a control method for an extended screen component provided in this application. This control method for the extended screen component can be applied in a server to generate a movement selection pointer based on hand key points to determine the target component, thereby improving the portability and accuracy of user selection of the extended screen component.

[0085] like Figure 5 As shown, the control method for the extended screen component further includes steps S301 to S304 before determining that the component to be judged intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be judged.

[0086] S301. Obtain the coordinates of the two endpoints of the preset edge of the component to be judged;

[0087] S302. Based on the coordinates of the first key point and the coordinates of the second key point, obtain the line segment expression corresponding to the moving selection pointer;

[0088] S303. Based on the two endpoint coordinates and the line segment expression, obtain the first value and the second value respectively;

[0089] S304. When the first value is negative and the second value is positive, or when the first value is positive and the second value is negative, determine that the extension line of the moving selection pointer intersects with the preset edge of the component to be judged.

[0090] In a specific embodiment, the line segment expression of the current moving selection pointer can be obtained based on the coordinates t1 of the first key point, t2 of the second key point, and the coordinates x0, y0, or t0 of the intersection point of the extension line of the moving selection pointer with the horizontal / vertical coordinate axes. For example, if the line segment expression is y = ax + b, substituting the coordinates t1 of the first key point, t2 of the second key point, and the coordinates x0, y0, or t0 of the intersection point can yield the values ​​of a and b, thus obtaining the line segment expression corresponding to the current moving selection pointer.

[0091] In a specific embodiment, the coordinates of the two endpoints of a preset edge are obtained. These coordinates are then substituted into the expression w = y - ax - b to obtain a first value w1 and a second value w2. The values ​​of w1 and w2 are compared. If both w1 and w2 are positive or both are negative, it indicates that the two endpoints of the preset edge are on the same side of the moving selection pointer, meaning the preset edge of the component to be judged does not intersect the extension of the current moving selection pointer. If w1 is positive and w2 is negative, or w1 is negative and w2 is positive, it indicates that the two endpoints of the preset edge are on opposite sides of the moving selection pointer, meaning the preset edge of the component to be judged intersects the extension of the current moving selection pointer. In this case, the component to be judged is determined to be the target component. The expression w = y - ax - b is used to determine whether a point is on the same side of the line segment y = ax + b, where the values ​​of a and b are the values ​​in the current moving selection pointer expression. Substituting the coordinates (x, y) of a point into the expression to obtain the value of w allows us to determine whether the point (x, y) is on the same side of the current line segment y = ax + b. For example, if the line segment expression of the current moving selection pointer is y = 2x + 1, then we get w = y - 2x - 1. The coordinates of the endpoints of the preset edge are (2, 1) and (1, 1). Substituting them into w = y - 2x - 1, we get w1 = -1 and w2 = -2. At this time, w1 and w2 are both negative, indicating that the endpoints of the preset edge are on the same side of the moving selection pointer, that is, the preset edge will not intersect with the extension of the moving selection pointer.

[0092] The above embodiments provide a control method, apparatus, computer device, and storage medium for an extended screen component. The method involves obtaining the coordinates of two endpoints of a preset edge of the component to be judged; obtaining a line segment expression corresponding to a moving selection pointer based on the coordinates of the first and second key points; obtaining a first value and a second value based on the two endpoint coordinates and the line segment expression; and determining that the extension line of the moving selection pointer intersects the preset edge of the component to be judged when the first value is negative and the second value is positive, or when the first value is positive and the second value is negative. This method, by obtaining the endpoint coordinates of the preset edge of the component to be judged and the line segment expression corresponding to the moving selection pointer, and by determining whether the two endpoints of the preset edge of the component to be judged are on the same side of the moving selection pointer, can accurately determine whether the preset edge of the component to be judged intersects the extension line of the moving selection pointer, thus improving the accuracy of target component determination.

[0093] Please see Figure 6 , Figure 6 This is a schematic block diagram of a control device for an extended screen assembly, provided in an embodiment of this application. This control device is used to execute the aforementioned control method for the extended screen assembly. The control device for the extended screen assembly can be configured on a server.

[0094] like Figure 6 As shown, the control device 400 for the extended screen assembly includes:

[0095] The screen outline recognition module 401 is used to recognize the screen outline of the user device in order to activate the screen extension mode.

[0096] The display interface module 402 is used to display the extended screen components and the screen of the user device in the display interface based on the relative positional relationship between each extended screen component and the screen outline.

[0097] The moving selection pointer generation module 403 is used to generate a moving selection pointer by tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm.

[0098] The target component determination module 404 is used to determine the extended screen component that intersects with the movement selection pointer as the target component when the user's hand is detected to move, so as to control the target component.

[0099] In one embodiment, the moving selection pointer generation module 403 includes:

[0100] The key point determination unit is used to identify the user's hand based on the preset algorithm and determine the first key point and the second key point;

[0101] A coordinate tracking unit is used to track the coordinates of the first key point of the first key point and the coordinates of the second key point of the second key point in the display interface when the user's hand is detected to be moving.

[0102] The moving selection pointer determination unit is used to obtain a line segment passing through the coordinates of the first key point and the second key point as the moving selection pointer, wherein the orientation of the moving selection pointer is from the first key point to the second key point.

[0103] In one embodiment, the target component determination module 404 includes:

[0104] The target area determination unit is used to determine the target area in the display interface based on the comparison result of the current slope of the movement selection pointer and the preset slope when the user's hand movement is detected.

[0105] The component to be judged determination unit is used to determine the extended screen component within the target area as the component to be judged.

[0106] The target component determination unit is used to determine that the component to be determined intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, and to designate the component to be determined as the target component.

[0107] In one embodiment, the target component determination module further includes a slope acquisition submodule, the slope acquisition submodule comprising:

[0108] The current slope acquisition unit is used to obtain the current slope of the movement selection pointer based on the coordinates of the first key point and the coordinates of the second key point;

[0109] The coordinate information acquisition unit is used to obtain the coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module.

[0110] The preset slope acquisition unit is used to determine the upper left corner and lower right corner of the display interface based on the coordinate information, and obtain the slope of the line segment passing through the upper left corner and the lower right corner as the preset slope.

[0111] In one embodiment, the target component determination module further includes an intersection determination submodule, the intersection determination submodule comprising:

[0112] An endpoint acquisition unit is used to acquire the coordinates of two endpoints of a preset edge of the component to be judged.

[0113] The line segment expression obtaining unit is used to obtain the line segment expression corresponding to the movement selection pointer based on the coordinates of the first key point and the coordinates of the second key point;

[0114] The numerical value acquisition unit is used to obtain a first numerical value and a second numerical value based on the coordinates of the two endpoints and the line segment expression, respectively.

[0115] An intersection determination unit is used to determine whether the extension line of the moving selection pointer intersects with a preset edge of the component to be judged when the first value is negative and the second value is positive, or when the first value is positive and the second value is negative.

[0116] In one embodiment, the control device 400 for the extended screen assembly further includes a position relationship acquisition module, the position relationship acquisition module comprising:

[0117] The first position obtaining unit is used to obtain the outline coordinates of the screen outline and, based on the outline coordinates, obtain the first position of the user device in the display interface.

[0118] The relative position relationship acquisition unit is used to obtain the relative position relationship between each of the extended screen components and the user device based on the preset size of each of the extended screen components, the screen size of the user device, and the first position.

[0119] In one embodiment, the control device 400 for the extended screen component further includes a component display module, the component display module comprising:

[0120] The corresponding page display unit is used to recognize the user's hand operation actions and, based on the operation actions, display the corresponding interface of the target component to the user.

[0121] In one embodiment, the corresponding page display unit includes:

[0122] An operation instruction determination subunit is used to identify the operation action, compare the operation action with the gesture actions in a preset gesture library, and determine the operation instruction corresponding to the operation action.

[0123] The details page display sub-unit is used to display the details page of the target component when the operation instruction is to display the details page;

[0124] The component switching subunit is used to switch the target component based on a preset component switching order when the operation instruction is component switching;

[0125] The display interface return sub-unit is used to exit the details page and return to the display interface when the operation instruction is to return to the display interface.

[0126] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0127] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 7 It runs on the computer device shown.

[0128] Please see Figure 7 , Figure 7 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0129] See Figure 7 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0130] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any control method for the extended screen components.

[0131] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0132] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any control method for the extended screen components.

[0133] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0135] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0136] Identify the screen outline of the user device to activate the screen extension mode;

[0137] Based on the relative positional relationship between each extended screen component and the screen outline, the extended screen components and the screen of the user device are displayed in the display interface;

[0138] Based on a preset algorithm, the coordinates of the first key point and the second key point of the user's hand in the display interface are tracked to generate a movement selection pointer.

[0139] When the user's hand movement is detected, the extended screen component that intersects with the movement selection pointer is identified as the target component, and the target component is controlled.

[0140] In one embodiment, when the processor generates a movement selection pointer by tracking the coordinates of a first key point and a second key point of the user's hand in the display interface based on a preset algorithm, it is used to:

[0141] Based on the preset algorithm, the user's hand is identified to determine the first key point and the second key point;

[0142] When the user's hand is detected to be moving, the coordinates of the first key point of the first key point and the coordinates of the second key point of the second key point are tracked in the display interface.

[0143] Obtain the line segment passing through the coordinates of the first key point and the second key point as the movement selection pointer, wherein the direction of the movement selection pointer is from the first key point to the second key point.

[0144] In one embodiment, when the processor determines the extended screen component intersecting the movement selection pointer as the target component upon detecting movement of the user's hand, it performs the following:

[0145] When the user's hand movement is detected, the target area is determined in the display interface based on the comparison result between the current slope of the movement selection pointer and the preset slope.

[0146] The extended screen component within the target area is taken as the component to be judged;

[0147] When the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, it is determined that the component to be judged intersects with the moving selection pointer, and the component to be judged is taken as the target component.

[0148] In one embodiment, before determining the target region in the display interface based on the comparison result of the current slope and the preset slope of the moving selection pointer, the processor is further configured to:

[0149] Based on the coordinates of the first key point and the second key point, the current slope of the moving selection pointer is obtained;

[0150] Based on the size information of the display interface acquired by the image acquisition module, the coordinate information of the display interface is obtained;

[0151] Based on the coordinate information, the top left corner and bottom right corner of the display interface are determined, and the slope of the line segment passing through the top left corner and bottom right corner is obtained as the preset slope.

[0152] In one embodiment, before determining that the component to be judged intersects the moving selection pointer when the extension of the moving selection pointer intersects the preset edge of the component to be judged, the processor is further configured to implement:

[0153] Obtain the coordinates of the two endpoints of the preset edge of the component to be judged;

[0154] Based on the coordinates of the first key point and the second key point, the line segment expression corresponding to the moving selection pointer is obtained;

[0155] Based on the two endpoint coordinates and the line segment expression, the first value and the second value are obtained respectively;

[0156] When the first value is negative and the second value is positive, or when the first value is positive and the second value is negative, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be judged.

[0157] In one embodiment, before displaying the extended screen components and the screen of the user device in the display interface based on the relative positional relationship between each extended screen component and the screen outline, the processor is further configured to:

[0158] Obtain the outline coordinates of the screen outline, and based on the outline coordinates, obtain the first position of the user device in the display interface;

[0159] Based on the preset dimensions of each of the extended screen components, the screen size of the user device, and the first position, the relative positional relationship between each of the extended screen components and the user device is obtained.

[0160] In one embodiment, after the processor determines the extended screen component intersecting the movement selection pointer as the target component when the user's hand movement is detected, it is further configured to:

[0161] The system identifies the user's hand gestures and, based on these gestures, displays the corresponding interface of the target component to the user.

[0162] In one embodiment, when the processor displays a corresponding interface of the target component to the user based on the operation action, it is configured to:

[0163] The operation action is identified, and the operation action is compared with the gesture actions in the preset gesture library to determine the operation instruction corresponding to the operation action.

[0164] When the operation instruction is to display the details page, the details page of the target component is displayed;

[0165] When the operation instruction is component switching, the target component is switched based on a preset component switching order;

[0166] When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

[0167] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the extended screen component control methods provided in the embodiments of this application.

[0168] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0169] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method for an extended screen component, characterized in that, include: Identify the screen outline of the user device to activate the screen extension mode; Based on the relative positional relationship between each extended screen component and the screen outline, the extended screen components and the screen of the user device are displayed in the display interface; Based on a preset algorithm, the coordinates of the first key point and the second key point of the user's hand in the display interface are tracked to generate a movement selection pointer. When the user's hand is detected to be moving, the extended screen component that intersects with the movement selection pointer is identified as the target component, and the target component is controlled. Wherein, the step of determining the extended screen component intersecting with the movement selection pointer as the target component when the user's hand movement is detected includes: When the user's hand movement is detected, the target area is determined in the display interface based on the comparison result between the current slope of the movement selection pointer and the preset slope. The extended screen component within the target area is taken as the component to be judged; When the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, it is determined that the component to be judged intersects with the moving selection pointer, and the component to be judged is taken as the target component.

2. The control method for the extended screen assembly according to claim 1, characterized in that, The step of tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm to generate a movement selection pointer includes: Based on the preset algorithm, the user's hand is identified to determine the first key point and the second key point; When the user's hand is detected to be moving, the coordinates of the first key point of the first key point and the coordinates of the second key point of the second key point are tracked in the display interface. Obtain the line segment passing through the coordinates of the first key point and the second key point as the movement selection pointer, wherein the direction of the movement selection pointer is from the first key point to the second key point.

3. The control method for the extended screen assembly according to claim 1, characterized in that, The comparison result between the current slope and the preset slope of the moving selection pointer, before determining the target area in the display interface, further includes: Based on the coordinates of the first key point and the second key point, the current slope of the moving selection pointer is obtained; Based on the size information of the display interface acquired by the image acquisition module, the coordinate information of the display interface is obtained; Based on the coordinate information, the top left corner and bottom right corner of the display interface are determined, and the slope of the line segment passing through the top left corner and bottom right corner is obtained as the preset slope.

4. The control method for the extended screen assembly according to claim 1, characterized in that, Before determining that the component to be judged intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, the method further includes: Obtain the coordinates of the two endpoints of the preset edge of the component to be judged; Based on the coordinates of the first key point and the second key point, the line segment expression corresponding to the moving selection pointer is obtained; Based on the two endpoint coordinates and the line segment expression, the first value and the second value are obtained respectively; When the first value is negative and the second value is positive, or when the first value is positive and the second value is negative, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be judged.

5. The control method for the extended screen assembly according to any one of claims 1-4, characterized in that, Before displaying the extended screen components and the user device screen in the display interface based on the relative positional relationship between each extended screen component and the screen outline, the method further includes: Obtain the outline coordinates of the screen outline, and based on the outline coordinates, obtain the first position of the user device in the display interface; Based on the preset dimensions of each of the extended screen components, the screen size of the user device, and the first position, the relative positional relationship between each of the extended screen components and the user device is obtained.

6. The control method for the extended screen assembly according to claim 1, characterized in that, After determining the extended screen component intersecting the movement selection pointer as the target component when the user's hand movement is detected, the method further includes: The system identifies the user's hand gestures and, based on these gestures, displays the corresponding interface of the target component to the user.

7. The control method for the extended screen assembly according to claim 6, characterized in that, The step of displaying the corresponding interface of the target component to the user based on the operation action includes: The operation action is identified, and the operation action is compared with the gesture actions in the preset gesture library to determine the operation instruction corresponding to the operation action. When the operation instruction is to display the details page, the details page of the target component is displayed; When the operation instruction is component switching, the target component is switched based on a preset component switching order; When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

8. A control device for an extended screen assembly, characterized in that, include: The screen outline recognition module is used to identify the screen outline of the user device in order to activate the screen extension mode. The display interface module is used to display the extended screen components and the screen of the user device in the display interface based on the relative positional relationship between each extended screen component and the screen outline. The moving selection pointer generation module is used to generate a moving selection pointer by tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm. The target component determination module is used to determine the extended screen component that intersects with the movement selection pointer as the target component when the user's hand is detected to move, so as to control the target component; The target component determination module includes: The target area determination unit is used to determine the target area in the display interface based on the comparison result of the current slope of the movement selection pointer and the preset slope when the user's hand movement is detected. The component to be judged determination unit is used to determine the extended screen component within the target area as the component to be judged. The target component determination unit is used to determine that the component to be determined intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, and to designate the component to be determined as the target component.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the control method for the extended screen component as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the control method for the extended screen assembly as described in any one of claims 1 to 7.

Images