Method and apparatus for controlling extension screen component, and computer device and storage medium

Abstract

Specifically disclosed in the present application are a method and apparatus for controlling an extension screen component, and a computer device and a storage medium. In the present application, a movable selection pointer is obtained by means of two key points of a hand of a user, the intersections between the movable selection pointer and extension screen components are then obtained when the user is moving the hand, and an extension screen component which intersects with the movable selection pointer is used as a target component. In this way, user operations are simplified, and the convenience and precision of a user selecting an extension screen component are improved.

Description

Control method, device, computer equipment and storage medium of extended screen assembly

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 27, 2023, with application number 2023112616808, and invention name “Control method, device, computer equipment and storage medium for extended screen assembly”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of smart wearable technology, and in particular to a control method, device, computer equipment, and storage medium for an extended screen assembly. Background Art

[0003] Currently, although there are some extended screen products for smart glasses (such as AR (Augmented Reality) glasses) and user devices (such as mobile phones, tablets, etc.) on the market, most of the operations on the extended screen of the user device use gestures or voice, which requires users to spend time learning and adapting to the operation methods. In addition, when the user's gestures are not clear enough and the language expression is unclear, the determined extended screen may not be what the user needs, and the accuracy of the extended screen selection is low. Therefore, how to improve the portability and accuracy of users' selection of extended screens has become an urgent problem that needs to be solved.

[0004] Summary of the Invention

[0005] The present application provides a control method, device, computer equipment and storage medium for an extended screen assembly to improve the portability of operation and the accuracy of extended screen selection.

[0006] In a first aspect, the present application provides a method for controlling an extended screen assembly, the method comprising:

[0007] Identify the screen outline of the user's device to activate screen extension mode;

[0008] Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface;

[0009] Tracking the coordinates of a first key point and a second key point of the user's hand on the display interface based on a preset algorithm to generate a moving selection pointer;

[0010] When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0011] In a second aspect, the present application further provides a control device for an extended screen assembly, the device comprising:

[0012] A screen outline recognition module, used to recognize the screen outline of the user device to activate the screen extension mode;

[0013] A display interface display module, configured to display the extended screen components and the screen of the user device in a display interface based on the relative positional relationship between each extended screen component and the screen outline;

[0014] a moving selection pointer generating module, configured to track the coordinates of a first key point and a second key point of a user's hand in the display interface based on a preset algorithm to generate a moving selection pointer;

[0015] The target component determination module is used to determine the extended screen component intersecting with the moving selection pointer as the target component when detecting the movement of the user's hand, so as to control the target component.

[0016] In a third aspect, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the control method of the extended screen component as described above when executing the computer program.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the control method of the extended screen assembly as described above.

[0018] The present application discloses a control method, device, computer equipment and storage medium for an extended screen component, which identifies the screen outline of a user device to activate the screen extension mode; based on the relative position relationship between each extended screen component and the screen outline, displays the extended screen component and the screen of the user device in a display interface; based on a preset algorithm, tracks the coordinates of the first key point and the second key point of the user's hand in the display interface to generate a mobile selection pointer; when the user's hand is detected to be moving, determines the extended screen component that intersects with the mobile selection pointer as a target component to control the target component. This method obtains the mobile selection pointer through the two key points of the user's hand, and then obtains the intersection of the mobile selection pointer and the extended screen component when the user moves his hand, and uses the extended screen component that intersects with the mobile selection pointer as the target component, which simplifies the user operation and improves the portability and accuracy of the user's selection of the extended screen component. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] FIG1 is a schematic flow chart of a first embodiment of a method for controlling an extended screen assembly provided by an embodiment of the present application;

[0021] FIG2 is a schematic diagram of a display interface of a control method for an extended screen assembly provided in an embodiment of the present application;

[0022] FIG3 is a schematic diagram showing the effect of adaptive changes in components of a control method for an extended screen component provided by an embodiment of the present application;

[0023] FIG4 is a schematic flow chart of a second embodiment of a method for controlling an extended screen assembly provided in an embodiment of the present application;

[0024] FIG5 is a schematic flow chart of a third embodiment of a method for controlling an extended screen assembly provided in an embodiment of the present application;

[0025] FIG6 is a schematic block diagram of a control device for an extended screen assembly provided in an embodiment of the present application;

[0026] FIG7 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0029] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should be further understood that the term “and / or” used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] Embodiments of the present application provide a control method, apparatus, computer device, and storage medium for an extended screen assembly. The control method for the extended screen assembly can be applied to a server, where a moving selection pointer is generated by hand key points to determine the target assembly, thereby improving the portability and accuracy of user selection of the extended screen assembly. The server can be a standalone server or a server cluster.

[0032] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0033] Please refer to Figure 1, which is a schematic flow chart of a first embodiment of a method for controlling an extended screen component provided by an embodiment of the present application. This method for controlling an extended screen component can be applied to a server to generate a moving selection pointer based on key points of a hand and then determine the target component, thereby improving the portability and accuracy of the user's selection of the extended screen component.

[0034] As shown in FIG1 , the control method of the extended screen assembly specifically includes steps S101 to S1034 .

[0035] S101: Identify the screen outline of a user device to activate a screen extension mode.

[0036] In one embodiment, when the screen of a user device (such as a mobile phone, tablet computer, etc.) is turned on by the user, the screen brightness will change.

[0037] In a specific embodiment, when a smart wearable device (such as AR glasses or an AR helmet) detects that the screen of a user device is on, it calls its own image acquisition module to identify the screen outline of the user device and activate the screen expansion mode. It is understood that the image acquisition module can be a camera built into the smart wearable device. In another embodiment, the image acquisition module can also be a camera connected to the smart wearable device.

[0038] In one embodiment, as shown in FIG2 , in screen extension mode, the user device and the extended screen component are in a bound state, and the display interface presented by the smart wearable device to the user will display the screen of the user device and the extended screen component of the user device or the smart wearable device.

[0039] It is understandable that the extended screen component can be an application component in the user device, which can display the application content of the user device, such as a video component that can display the playing video.

[0040] In another embodiment, the extended screen component may also be a functional component in the smart wearable device, which may display relevant functions of the smart wearable device so that users can query and operate it.

[0041] Among them, the extended screen component is a high-performance cross-terminal development framework for smart large-screen devices, a fast application engine running on smart wearable devices, which can help developers quickly build application services for smart wearable devices and provide multi-screen interaction capabilities.

[0042] S102: Based on the relative positional relationship between each extended screen component and the screen outline, display the extended screen components and the screen of the user device in a display interface.

[0043] Based on the relative positional relationship between each extended screen component and the screen outline, before displaying the extended screen component and the screen of the user device in the display interface, it also includes: obtaining the outline coordinates of the screen outline, and based on the outline coordinates, obtaining the first position of the user device in the display interface; based on the preset size of each extended screen component, the screen size of the user device and the first position, obtaining the relative positional relationship between each extended screen component and the user device.

[0044] In one embodiment, the smart wearable device invokes an image acquisition module to identify the outline of the user device's screen and simultaneously record the outline coordinates of the user device's screen. The coordinates of the upper left corner and lower right corner of the user device's screen are then filtered from the outline coordinates to represent the first position of the user device's screen on the display interface. For example, after a user turns on a mobile phone screen, the AR glasses identify the outline of the phone screen and obtain the coordinates of the upper left corner and lower right corner of the phone screen outline to represent the first position of the phone screen.

[0045] In one embodiment, a smart wearable device (such as smart glasses, etc.) projects the screen content of the user device or the related content of the smart wearable device itself as AR content in the form of an extended screen component and displays it in the external space of the screen outline, forming an extended display of the user device screen.

[0046] In one embodiment, the display interface is an interface displayed by the smart wearable device to the user for viewing, and is captured by the image acquisition module of the smart wearable device. It is understandable that, when the image captured by the image acquisition module is not cropped, the size of the image captured by the image acquisition module is the same as the size of the display interface. The image size can be calculated based on the preset image resolution and pixel value of the image acquisition device. For example, if the pixels of a picture are 600×800 pixels and the resolution is 300×400, then the actual length and width are: 600 / 300=2 inches, 800 / 400=2 inches, and the length and width are both approximately 5 cm, that is, the picture is actually 5 cm long and 5 cm wide. The coordinate information of the display interface can be obtained based on the image size information captured by the image acquisition module, and the coordinate system can be determined based on the image size information captured by the image acquisition module. For example, when the image acquisition module is a camera device, it is known that the width of the image captured by the camera is c w , with a height of c h , take the upper left corner as the origin of the coordinate system (0, 0), the leftmost side of the display interface as the vertical axis of the coordinate system, and the top of the display interface as the horizontal axis, then the coordinates of the lower right corner of the display interface can be obtained as (c w , c h ).

[0047] In one embodiment, the relative positional relationship between the extended screen components and the first position of the user device is determined based on the size of each extended screen component and the screen size of the user device. For example, the user device is a mobile phone with a screen size of 7.5 cm wide and 13.5 cm long. The extended screen components have the following dimensions: first extended screen component: 2 cm x 2 cm, second component: 1.5 cm x 1.5 cm, third component: 1.5 cm x 2 cm, etc. Based on the width and length of the mobile phone screen, appropriately sized extended screen components can be arranged in rows at the top of the phone and in columns on the sides of the phone to determine the relative positional relationship between the extended screen components and the phone screen. For example, the first extended screen component is located in the first column and second row on the left side of the phone screen.

[0048] In one embodiment, the position coordinates of each extended screen component are determined based on the relative positional relationship between each extended screen component and the first position of the user device and the size of the extended screen component. For example, the upper left corner point of the user device is (20, 30) and the lower right corner point is (30, 50), then the first position is expressed as ((20, 30), (30, 50)), the size of the first extended screen component is 2 cm wide and 2 cm long, and the relative positional relationship with the user device is that the first extended screen component is located in the first row and first column at the top of the user device, then the coordinates of the upper left corner point of the first extended screen component can be obtained as (20, 28), and the lower right corner point is (22, 32), and the position coordinates of the first extended screen component can be expressed as T1(20, 28), (22, 32).

[0049] The screen size of the user device is known and can be obtained by measurement or from the product information of the user device. The size of each extended screen component can be set by the user or preset by the developer.

[0050] The positions of the various extended screen components are shown as follows:

[0051] Here, x1y1 represents the coordinates of the upper left corner of each extended screen component, x2y2 represents the coordinates of the lower right corner of each extended screen component, and T1, T2, ..., Tn represent the positions of the 1st, 2nd, ...nth extended screen components. That is, the coordinates of the upper left corner and lower right corner of the extended screen component represent the position of the extended screen component.

[0052] In one embodiment, based on the obtained positions of each extended screen component in the display interface and the first position of the screen outline of the user device in the display interface, the each extended screen component is displayed around the screen of the user device. For example, as shown in Figure 2, the each extended screen component is distributed on the left and right sides and top of the screen of the user device.

[0053] S103 : Based on a preset algorithm, track the coordinates of a first key point and a second key point of the user's hand in the display interface to generate a moving selection pointer.

[0054] Based on a preset algorithm, the coordinates of the first key point and the coordinates of the second key point of the user's hand in the display interface are tracked to generate a moving selection pointer, including: based on the preset algorithm, identifying the user's hand and determining the first key point and the second key point; when the user's hand is detected to be moving, tracking the first key point coordinates of the first key point and the second key point coordinates of the second key point in the display interface; obtaining a line segment passing through the first key point coordinates and the second key point coordinates as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0055] In one embodiment, the preset algorithm is a hand key point algorithm, which can track the coordinates of the hand key points in real time based on the movement of the user's hand. For example, a hand key point detection method based on MediaPipe (MediaPipe is a framework for building cross-platform machine learning solutions) includes a palm detector and a hand coordinate model. The palm detector is used to determine the position of the user's hand, and the hand coordinate model is used to obtain the coordinates of the hand key points. When a human hand is detected, the coordinates of the human hand key points can be tracked in real time as the hand moves.

[0056] In a specific embodiment, a preset algorithm is stored in the smart wearable device. When in screen expansion mode, the smart wearable device calls the preset algorithm to identify 21 key points of the hand, and determines the first key point and the second key point among the key points. For example, the first joint of the user's thumb is used as the first key point, and the fingertip of the user's thumb is used as the second key point.

[0057] In one embodiment, the movement of the user's hand may cause the coordinates of the first key point and the second key point to change. When the change of the coordinates of the first key point and the second key point is detected, it can be determined that the user's hand is moving.

[0058] In one embodiment, when the user's hand moves, the first key point coordinates corresponding to the first key point of the user's hand and the second key point coordinates corresponding to the second key point are tracked in real time through a preset algorithm, and the line segment passing through the first key point coordinates and the second key point coordinates is used as a moving selection pointer.

[0059] In one embodiment, the direction of the mobile selection pointer points from the first key point to the second key point. For example, a line segment starting from the first joint of the thumb and connecting the tip of the thumb is the mobile selection pointer, and the direction of the first joint of the thumb pointing to the fingertip is the direction of the mobile selection pointer.

[0060] S104: When the user's hand is detected to be moving, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0061] When the movement of the user's hand is detected, after determining the extended screen component that intersects with the mobile selection pointer as the target component, it also includes: identifying the operation action of the user's hand, and based on the operation action, displaying the corresponding interface of the target component to the user.

[0062] Based on the operation action, the corresponding interface of the target component is displayed to the user, including: identifying the operation action, comparing the operation action with the gesture actions in a preset gesture library, and determining the operation instruction corresponding to the operation action; when the operation instruction is to display the details page, the details page of the target component is displayed; when the operation instruction is to switch components, the target component is switched based on a preset component switching order; when the operation instruction is to return to the display interface, exit the details page and return to the display interface.

[0063] In one embodiment, as the user moves their hand, the mobile selection pointer changes accordingly. For example, if the user changes the direction of their thumb, the direction of the mobile selection pointer generated by the user's thumb changes accordingly. During the movement process, the mobile selection pointer may intersect with the extended screen components surrounding the user device screen in the display interface, and the extended screen component intersected by the mobile selection pointer is used as the target component.

[0064] In one embodiment, after the target component is determined, the display effect of the preset component is selected, and the corresponding display instructions are generated to perform component adaptive changes on the target component, for example, changes in component size and transparency. As shown in Figure 3, after the target component is selected, the target component will become larger to achieve a similar floating effect. At the same time, the transparency of other unselected components will increase to highlight the target component.

[0065] In one embodiment, after determining the target component, the user's hand operation is identified and compared with the gestures in a preset gesture library. The gesture library includes gestures and corresponding operation instructions. After determining the gesture corresponding to the operation, the corresponding operation instruction can be obtained. The operation instruction includes displaying a details page, switching components, returning to the main interface (i.e., the display interface), etc. The switching components can be switched according to a preset component switching order.

[0066] The above embodiment provides a control method, device, computer equipment and storage medium for an extended screen component, which identifies the screen outline of a user device to activate the screen extension mode; based on the relative position relationship between each extended screen component and the screen outline, displays the extended screen component and the screen of the user device in a display interface; based on a preset algorithm, tracks the coordinates of the first key point and the second key point of the user's hand in the display interface to generate a mobile selection pointer; when the user's hand is detected to be moving, determines the extended screen component that intersects with the mobile selection pointer as a target component to control the target component. This method obtains the mobile selection pointer through the two key points of the user's hand, and then obtains the intersection of the mobile selection pointer and the extended screen component when the user moves his hand. The extended screen component that intersects with the mobile selection pointer is used as the target component, which simplifies the user operation and improves the portability and accuracy of the user's selection of the extended screen component.

[0067] Please refer to Figure 4, which is a schematic flow chart of a second embodiment of a method for controlling an extended screen component provided by an embodiment of the present application. This method for controlling an extended screen component can be applied to a server to generate a moving selection pointer based on key points of a hand and then determine the target component, thereby improving the portability and accuracy of the user's selection of the extended screen component.

[0068] As shown in FIG. 4 , step S104 of the control method of the extended screen assembly specifically includes steps S201 to S203 .

[0069] S201: When detecting movement of the user's hand, determining a target area in the display interface based on a comparison result of a current slope of the moving selection pointer and a preset slope;

[0070] S202: taking the extended screen component in the target area as a component to be determined;

[0071] S203: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, determine that the component to be determined intersects with the moving selection pointer, and use the component to be determined as the target component.

[0072] Based on the comparison result of the current slope of the mobile selection pointer and the preset slope, before determining the target area in the display interface, it also includes: obtaining the current slope of the mobile selection pointer based on the first key point coordinates and the second key point coordinates; obtaining the coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module; determining the upper left corner point and the lower right corner point of the display interface based on the coordinate information, and obtaining the slope of the line segment passing through the upper left corner point and the lower right corner point as the preset slope.

[0073] In a specific embodiment, the display interface is an interface displayed by the smart wearable device to the user, and is captured by the image acquisition module of the smart wearable device. It is understandable that, when the image captured by the image acquisition module is not cropped, the size of the image captured by the image acquisition module is the same as the size of the display interface. The size information of the display interface can be obtained based on the image size information captured by the image acquisition module, and the coordinate system can be determined based on the image size information captured by the image acquisition module. For example, when the image acquisition module is a camera device, it is known that the width of the image captured by the camera is c w , with a height of c h , take the upper left corner as the origin of the coordinate system (0, 0), the leftmost side of the display interface as the vertical axis of the coordinate system, and the top of the display interface as the horizontal axis, then the coordinates of the lower right corner of the display interface can be obtained as (c w , c h ), the slope of the line segment formed by the upper left corner point and the lower right corner point is c k , c k as the preset slope.

[0074] In a specific embodiment, the coordinates of the first key point are set to t1, and the coordinates of the second key point are set to t2. The current slope t of the moving selection pointer can be obtained based on the two coordinate points t1 and t2. k .

[0075] In one embodiment, the current slope t of the moving selection pointer is k With the preset slope c k By comparing, the area where the target component is located can be preliminarily determined and the area is taken as the target area. All the extended screen components in the target area are taken as components to be determined whether they intersect with the extension line of the moving selection pointer.

[0076] In a specific embodiment, if t k <c k , then extend the line segment t2t1 to the leftmost point x=0 on the display interface, obtaining the intersection point x0 with the leftmost edge of the display interface, and obtain the line segment passing through t2, t1, and x0. That is, the extension line of the moving selection pointer will pass through the left area of ​​the user device and intersect with the preset edge of a certain extended screen component in the left area of ​​the user device. At this time, the left area is used as the target area. As shown in the first figure in Figure 3, at this time, the current slope t of the moving selection pointer is k Less than the slope of the display interface (i.e. the preset slope) c k , the extension line of the moving selection pointer intersects with one of the extended screen components in the left area of ​​the user device screen in the display interface, then the left area is used as the target area, and all the extended screen components in the left area are used as components to be judged.

[0077] In one embodiment, if t k >c k , then extend the line segment t2t1 to the top edge of the display interface at y=0, obtain the intersection y0 with the top edge of the display interface, and obtain the line segment passing through t2, t1, and y0. That is, the extension line of the moving selection pointer will pass through the top area of ​​the user device and intersect with the preset edge of a certain extended screen component in the top area of ​​the user device. At this time, the top area is used as the target area. As shown in the third figure in Figure 3, at this time, the current slope t of the moving selection pointer is k Greater than the slope of the display interface (i.e. the preset slope) c k , the extension line of the moving selection pointer intersects with one of the extended screen components in the top area of ​​the user device screen in the display interface, then the top area is used as the target area, and all extended screen components in the top area are used as components to be judged.

[0078] In one embodiment, if t k =c k, then extend the line segment t2t1 to the upper left corner origin (0,0), obtain the intersection t0 with the upper left corner of the display interface, and obtain the line segment passing through t2, t1, and t0. That is, the extension line of the moving selection pointer will pass through the upper left corner area of ​​the user device and intersect with the preset edge of a certain extended screen component in the upper left corner area of ​​the user device. At this time, the upper left corner area is used as the target area. As shown in the second figure in Figure 3, at this time, the current slope t of the moving selection pointer k Equal to the slope of the display interface (ie, the preset slope) c k , the extension line of the moving selection pointer intersects with the extended screen component that exists in the upper left corner area of ​​the user device screen in the display interface, then the upper left corner area is used as the target area, and the extended screen component in the upper left corner area is used as the component to be judged.

[0079] In one embodiment, when the user uses his left hand to operate, the finger may point to the right area of ​​the user device, and the extension line of the moving selection pointer has a chance to intersect with the preset edge of the extended screen assembly in the right area.

[0080] In one embodiment, the side of the component to be judged close to the user device screen is used as the preset side. For example, when the left area of ​​the user device is determined to be the target area, the extended screen component in the left area is used as the component to be judged, and the right border of the component to be judged is used as the preset side; when the right area of ​​the user device is determined to be the target area, the extended screen component in the right area is used as the component to be judged, and the left border of the component to be judged is used as the preset side; when the top area of ​​the user device is determined to be the target area, the extended screen component in the top area is used as the component to be judged, and the bottom border of the component to be judged is used as the preset side.

[0081] When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the moving selection pointer, and the component to be determined is used as the target component.

[0082] It can be understood that when determining the target component, first, based on the relationship between the slope of the mobile selection pointer and the preset slope, the target area where the target component may be located is determined, and all the extended screen components in the target area are taken as components to be determined whether they are the target components, and then the edge of the component to be determined adjacent to the user device screen is taken as the preset edge, and whether the extension line of the mobile selection pointer intersects with the preset edge can be determined to determine a unique target component. For example, as shown in the second figure in Figure 3, the slope of the mobile selection pointer is equal to the preset slope. At this time, the upper left corner area of ​​the user device screen is taken as the target area, and the component in the upper left corner is taken as the component to be determined, and the target component is determined in the components to be determined. Even if the mobile selection pointer also intersects with the extended screen component in the left area, since the left area is not the target area, the extended screen component in the left area has been excluded, so the mobile selection pointer only intersects with the extended screen component in the upper left corner.

[0083] The above embodiment provides a control method, device, computer equipment and storage medium for an extended screen component. When the user's hand is detected to be moving, a target area is determined in the display interface based on the comparison result of the current slope of the mobile selection pointer and the preset slope; the extended screen component in the target area is used as the component to be determined; when the extension line of the mobile selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the mobile selection pointer, and the component to be determined is used as the target component. This method determines the target slope by comparing the current slope of the mobile selection pointer with the preset slope, obtains the extended screen component in the target area as the component to be determined, and accurately determines the extended screen component that intersects with the mobile selection pointer as the target component by judging whether the preset edge of the component to be determined intersects with the extension line corresponding to the mobile selection pointer, thereby improving the accuracy of the target component determination.

[0084] Please refer to Figure 5, which is a schematic flow chart of a third embodiment of a method for controlling an extended screen component provided in an embodiment of the present application. This method for controlling an extended screen component can be applied to a server to generate a moving selection pointer based on key points of a hand and then determine the target component, thereby improving the portability and accuracy of the user's selection of an extended screen component.

[0085] As shown in Figure 5, the steps of the control method of the extended screen component include steps S301 to S304 before determining that the component to be judged intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be judged.

[0086] S301, obtaining the coordinates of two endpoints of the preset edge of the component to be determined;

[0087] S302: Based on the first key point coordinates and the second key point coordinates, obtain a line segment expression corresponding to the moving selection pointer;

[0088] S303, obtaining a first value and a second value based on the two endpoint coordinates and the line segment expression;

[0089] S304: When the first value is negative and the second value is positive, or when the first value is positive and the second value is negative, determine whether the extension line of the moving selection pointer intersects with a preset edge of the component to be determined.

[0090] In a specific embodiment, a line segment expression for the current moving selection pointer can be obtained based on the coordinates t1 of the first key point of the moving selection pointer, the coordinates t2 of the second key point, and the coordinates x0, y0, or t0 of the intersection of the extended line of the moving selection pointer and the horizontal / vertical coordinate axis. For example, assuming the line segment expression is y = ax + b, substituting the first key point coordinate t1, the second key point coordinate t2, and the intersection coordinates x0, y0, or t0 into the line segment expression, the values ​​of a and b can be obtained to obtain the line segment expression corresponding to the current moving selection pointer.

[0091] In a specific embodiment, the endpoint coordinates of the two endpoints of the preset edge are obtained and substituted into the expression w = y-ax-b to obtain a first value w1 and a second value w2. The values ​​of w1 and w2 are compared. When w1 and w2 are both positive or negative, it indicates that the two endpoints of the preset edge are on the same side of the moving selection pointer, that is, the preset edge of the current component to be determined does not intersect with the extension line of the current moving selection pointer. When w1 is positive and w2 is negative, or w1 is negative and w2 is positive, it indicates that the two endpoints of the preset edge are on both sides of the moving selection pointer, that is, the preset edge of the current component to be determined intersects with the extension line of the current moving selection pointer, and the component to be determined is determined to be the target component. The expression w = y-ax-b is used to determine whether a point is on the same side of the line segment y = ax + b, where the values ​​of a and b are the values ​​in the expression for the current moving selection pointer. Substituting the coordinates (x, y) of a point into the expression to obtain the value of w can determine whether the point (x, y) is on the same side of the current line segment y = ax + b. For example, the line segment expression of the current moving selection pointer is y=2x+1, then w=y-2x-1 is obtained, and the endpoint coordinates of the preset edge are (2,1) and (1,1). Substituting into w=y-2x-1, we get w1=-1, w2=-2. At this time, w1 and w2 are both negative numbers, indicating that the endpoints of the preset edge are on the same side of the moving selection pointer, that is, the preset edge will not intersect with the extension line of the moving selection pointer.

[0092] The above embodiment provides a control method, device, computer equipment and storage medium for an extended screen component, which obtains the coordinates of the two endpoints of the preset edge of the component to be judged; based on the first key point coordinates and the second key point coordinates, obtains the line segment expression corresponding to the mobile selection pointer; based on the two endpoint coordinates and the line segment expression, obtains a first numerical value and a second numerical value respectively; when the first numerical value is negative and the second numerical value is positive, or the first numerical value is positive and the second numerical value is negative, determines that the extension line of the mobile selection pointer intersects with the preset edge of the component to be judged. This method obtains the endpoint coordinates of the preset edge of the component to be judged and the line segment expression corresponding to the mobile selection pointer, and determines whether the two endpoints of the preset edge of the component to be judged are on the same side of the mobile selection pointer, thereby improving the accuracy of target component determination.

[0093] Please refer to Figure 6, which is a schematic block diagram of a control device for an extended screen assembly according to an embodiment of the present application, wherein the control device for the extended screen assembly is used to execute the aforementioned control method for the extended screen assembly.

[0094] As shown in FIG6 , the control device 400 of the extended screen assembly includes:

[0095] A screen outline recognition module 401 is used to recognize the screen outline of the user device to activate the screen extension mode;

[0096] A display interface display module 402, configured to display the extended screen components and the screen of the user device in a display interface based on the relative positional relationship between each extended screen component and the screen outline;

[0097] A moving selection pointer generating module 403 is configured to track the coordinates of a first key point and a second key point of the user's hand in the display interface based on a preset algorithm to generate a moving selection pointer;

[0098] The target component determination module 404 is configured to determine the extended screen component intersecting with the moving selection pointer as a target component when detecting movement of the user's hand, so as to control the target component.

[0099] In one embodiment, the mobile selection pointer generation module 403 includes:

[0100] A key point determination unit, configured to identify the user's hand based on the preset algorithm and determine a first key point and a second key point;

[0101] a coordinate tracking unit, configured to track a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface when detecting movement of the user's hand;

[0102] A moving selection pointer determining unit is used to obtain a line segment passing through the coordinates of the first key point and the coordinates of the second key point as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0103] In one embodiment, the target component determination module 404 includes:

[0104] a target area determination unit, configured to determine a target area in the display interface based on a comparison result of a current slope of the moving selection pointer and a preset slope when detecting movement of the user's hand;

[0105] a component-to-be-judged determining unit, configured to take the extended screen component in the target area as the component to be judged;

[0106] A target component determining unit is configured to determine that the component to be determined intersects with the moving selection pointer when an extension line of the moving selection pointer intersects with a preset edge of the component to be determined, and to use the component to be determined as the target component.

[0107] In one embodiment, the target component determination module further includes a slope acquisition submodule, and the slope acquisition submodule includes:

[0108] a current slope obtaining unit, configured to obtain a current slope of the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0109] A coordinate information obtaining unit, configured to obtain coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module;

[0110] The preset slope obtaining unit is used to determine the upper left corner point and the lower right corner point of the display interface based on the coordinate information, and obtain the slope of the line segment passing through the upper left corner point and the lower right corner point as the preset slope.

[0111] In one embodiment, the target component determination module further includes an intersection determination submodule, and the intersection determination submodule includes:

[0112] An endpoint acquisition unit, configured to acquire the coordinates of two endpoints of a preset edge of the component to be determined;

[0113] a line segment expression obtaining unit, configured to obtain a line segment expression corresponding to the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0114] a value obtaining unit, configured to obtain a first value and a second value based on the two endpoint coordinates and the line segment expression;

[0115] An intersection determination unit is used to determine whether the extension line of the moving selection pointer intersects with the preset edge of the component to be judged when the first value is negative and the second value is positive, or when the first value is positive and the second value is negative.

[0116] In one embodiment, the control device 400 of the extended screen assembly further includes a position relationship obtaining module, and the position relationship obtaining module includes:

[0117] a first position obtaining unit, configured to obtain outline coordinates of the screen outline, and obtain a first position of the user equipment in the display interface based on the outline coordinates;

[0118] A relative position relationship obtaining unit is used to obtain the relative position relationship between each of the extended screen components and the user equipment based on the preset size of each of the extended screen components, the screen size of the user equipment and the first position.

[0119] In one embodiment, the control device 400 for the extended screen assembly further includes a component display module, wherein the component display module includes:

[0120] The corresponding page display unit is used to identify the operation action of the user's hand and display the corresponding interface of the target component to the user based on the operation action.

[0121] In one embodiment, the corresponding page display unit includes:

[0122] An operation instruction determination subunit, configured to identify the operation action, compare the operation action with gesture actions in a preset gesture library, and determine an operation instruction corresponding to the operation action;

[0123] A details page display subunit, configured to display the details page of the target component when the operation instruction is to display the details page;

[0124] a component switching subunit, configured to switch the target component based on a preset component switching sequence when the operation instruction is a component switching;

[0125] The display interface returning subunit is configured to exit the details page and return to the display interface when the operation instruction is to return to the display interface.

[0126] It should be noted that those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] The above-mentioned apparatus may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in FIG. 7 .

[0128] Please refer to Figure 7, which is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0129] 7 , the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0130] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can enable the processor to execute any one of the control methods for the extended screen assembly.

[0131] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0132] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any control method of the extended screen component.

[0133] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that the structure shown in FIG7 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0136] Identify the screen outline of the user's device to activate screen extension mode;

[0137] Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface;

[0138] Tracking the coordinates of a first key point and a second key point of the user's hand on the display interface based on a preset algorithm to generate a moving selection pointer;

[0139] When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0140] In one embodiment, when the processor tracks the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm and generates a moving selection pointer, it is configured to implement:

[0141] Based on the preset algorithm, the user's hand is identified to determine a first key point and a second key point;

[0142] When detecting movement of the user's hand, tracking a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface;

[0143] A line segment passing through the coordinates of the first key point and the coordinates of the second key point is obtained as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0144] In one embodiment, when the processor detects movement of the user's hand and determines the extended screen component intersecting with the moving selection pointer as the target component, it is configured to implement:

[0145] When the user's hand is detected to be moving, determining a target area in the display interface based on a comparison result of a current slope of the moving selection pointer and a preset slope;

[0146] Taking the extended screen component in the target area as the component to be determined;

[0147] When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the moving selection pointer, and the component to be determined is used as the target component.

[0148] In one embodiment, before determining the target area in the display interface based on the comparison result of the current slope of the moving selection pointer and the preset slope, the processor is further configured to implement:

[0149] Obtaining a current slope of the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0150] Obtaining coordinate information of the display interface based on size information of the display interface acquired by the image acquisition module;

[0151] Based on the coordinate information, the upper left corner point and the lower right corner point of the display interface are determined, and the slope of the line segment passing through the upper left corner point and the lower right corner point is obtained as the preset slope.

[0152] In one embodiment, when the extended line of the moving selection pointer intersects the preset edge of the component to be determined, the processor is further configured to implement, before determining that the component to be determined intersects the moving selection pointer:

[0153] Obtain the coordinates of two endpoints of the preset edge of the component to be judged;

[0154] Based on the first key point coordinates and the second key point coordinates, obtaining a line segment expression corresponding to the moving selection pointer;

[0155] Based on the two endpoint coordinates and the line segment expression, respectively obtain a first value and a second value;

[0156] When the first value is a negative number and the second value is a positive number, or when the first value is a positive number and the second value is a negative number, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be determined.

[0157] In one embodiment, the processor is further configured to implement, before displaying the extended screen components and the screen of the user device in the display interface based on the relative positional relationship between each extended screen component and the screen outline:

[0158] Acquire outline coordinates of the screen outline, and obtain a first position of the user equipment in the display interface based on the outline coordinates;

[0159] Based on the preset size of each extended screen component, the screen size of the user equipment and the first position, the relative position relationship between each extended screen component and the user equipment is obtained.

[0160] In one embodiment, after determining the extended screen component intersecting with the moving selection pointer as the target component when detecting the movement of the user's hand, the processor is further configured to implement:

[0161] Identify the operation action of the user's hand, and based on the operation action, display the corresponding interface of the target component to the user.

[0162] In one embodiment, when the processor displays the corresponding interface of the target component to the user based on the operation action, the processor is configured to implement:

[0163] Identifying the operation action, comparing the operation action with gesture actions in a preset gesture library, and determining an operation instruction corresponding to the operation action;

[0164] When the operation instruction is to display a details page, display the details page of the target component;

[0165] When the operation instruction is a component switching, switching the target component based on a preset component switching order;

[0166] When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

[0167] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any control method of the extended screen assembly provided in the embodiment of the present application.

[0168] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device.

[0169] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A control method for an extended screen assembly, comprising: Identify the screen outline of the user's device to activate the screen extension mode; Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface; Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a moving selection pointer; When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

2. The control method of the extended screen assembly according to claim 1, wherein: The step of tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm to generate a moving selection pointer includes: Based on the preset algorithm, the user's hand is identified to determine a first key point and a second key point; When detecting that the user's hand moves, tracking a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface; A line segment passing through the coordinates of the first key point and the coordinates of the second key point is obtained as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

3. The control method of the extended screen assembly according to claim 2, wherein: The first key point includes a first joint of the thumb, the second key point includes a tip of the thumb, and the direction of the moving selection pointer includes a direction in which the first joint of the thumb points to the tip of the thumb.

4. The control method of the extended screen assembly according to claim 1, wherein: The step of determining, when detecting that the user's hand is moving, an extended screen component intersecting with the moving selection pointer as a target component comprises: When the user's hand is detected to be moving, based on a comparison result between a current slope of the moving selection pointer and a preset slope, a target area is determined in the display interface; Taking the extended screen component in the target area as the component to be determined; When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, the The component to be determined intersects with the mobile selection pointer, and the component to be determined is used as the target component.

5. The control method of the extended screen assembly according to claim 4, wherein: The determining of the target area in the display interface based on the comparison result of the current slope of the moving selection pointer and the preset slope includes: When the comparison result is that the current slope is less than the preset slope, a first area in the display interface is determined as the target area, wherein the first area includes a left area of ​​the user equipment.

6. The control method of the extended screen assembly according to claim 5, wherein: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer and taking the component to be determined as the target component, the method further includes: Extending the mobile selection pointer according to the direction until it intersects with the left edge of the display interface, thereby obtaining an extension line of the mobile selection pointer; The right border of the component to be judged is used as the preset border.

7. The control method of the extended screen assembly according to claim 4, wherein: The step of determining a target area in the display interface based on a comparison result of the current slope of the moving selection pointer and a preset slope further includes: When the comparison result is that the current slope is greater than the preset slope, a second area in the display interface is determined as the target area, wherein the second area includes a top area of ​​the user equipment.

8. The control method of the extended screen assembly according to claim 7, wherein: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer and taking the component to be determined as the target component, the method further includes: Extending the moving selection pointer according to the direction until it intersects with the upper edge of the display interface, thereby obtaining an extension line of the moving pointer; The lower border of the component to be determined is used as the preset border.

9. The control method of the extended screen assembly according to claim 4, wherein: The step of determining a target area in the display interface based on a comparison result of the current slope of the moving selection pointer and a preset slope further includes: When the comparison result is that the current slope is equal to the preset slope, A third area is determined as the target area, wherein the third area includes an upper left corner area of ​​the user equipment.

10. The control method of the extended screen assembly according to claim 9, wherein: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer and taking the component to be determined as the target component, the method further includes: Extending the moving selection pointer according to the direction until it intersects with the upper left corner point of the display interface, thereby obtaining an extension line of the moving pointer; The bottom border and the right border of the component to be judged are used as the preset borders.

11. The control method of the extended screen assembly according to claim 4, wherein: The method further includes, before determining the target area in the display interface based on the comparison result of the current slope of the moving selection pointer and the preset slope: Based on the first key point coordinates and the second key point coordinates, obtaining a current slope of the moving selection pointer; Based on the size information of the display interface acquired by the image acquisition module, obtaining the coordinate information of the display interface; Based on the coordinate information, the upper left corner point and the lower right corner point of the display interface are determined, and the slope of the line segment passing through the upper left corner point and the lower right corner point is obtained as the preset slope.

12. The control method of the extended screen assembly according to claim 4, wherein: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer, the method further includes: Obtain the coordinates of two endpoints of the preset edge of the component to be determined; Based on the first key point coordinates and the second key point coordinates, obtaining a line segment expression corresponding to the moving selection pointer; Based on the two endpoint coordinates and the line segment expression, respectively obtain a first value and a second value; When the first value is a negative number and the second value is a positive number, or when the first value is a positive number and the second value is a negative number, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be determined.

13. The control method of the extended screen assembly according to claim 1, wherein: Based on the relative positional relationship between each extended screen component and the screen outline, before displaying the extended screen component and the screen of the user equipment in the display interface, the method further includes: Acquire contour coordinates of the screen contour, and obtain a first position of the user equipment in the display interface based on the contour coordinates; Based on the preset size of each of the extended screen components, the screen size of the user equipment and the first position, the relative position relationship between each of the extended screen components and the user equipment is obtained.

14. The control method of the extended screen assembly according to claim 1, wherein: After the extended screen component intersecting with the moving selection pointer is determined as a target component when the user's hand is detected to be moving, the method further includes: Identify the operation action of the user's hand, and based on the operation action, display the corresponding interface of the target component to the user.

15. The control method of the extended screen assembly according to claim 14, wherein: The displaying the corresponding interface of the target component to the user based on the operation action includes: Identify the operation action, compare the operation action with gesture actions in a preset gesture library, and determine the operation instruction corresponding to the operation action; When the operation instruction is to display a detail page, display the detail page of the target component; When the operation instruction is a component switching, switching the target component based on a preset component switching sequence; When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

16. The control method of the extended screen assembly according to claim 1, wherein: The method of displaying the extended screen components and the screen of the user device in a display interface based on the relative position relationship between each extended screen component and the screen outline includes: Based on the first position of the user equipment in the display interface, the size of each of the extended screen components and the relative position relationship of the user position, obtaining the position of each of the extended screen components in the display interface; Based on the position of each of the extended screen components in the display interface and the first position, the extended screen components and the screen of the user device are displayed in the display interface.

17. The control method of the extended screen assembly according to claim 1, wherein: After the extended screen component intersecting with the moving selection pointer is determined as a target component when the user's hand is detected to be moving, the method further includes: Based on a preset component selection effect, generating a display instruction corresponding to the component selection effect; Based on the display instruction, the target component is controlled to perform display changes.

18. A control device for an extended screen assembly, comprising: A screen outline recognition module, used to recognize the screen outline of the user device to activate the screen extension mode; A display interface display module, configured to display the extended screen components and the screen of the user device in a display interface based on the relative positional relationship between each extended screen component and the screen outline; A moving selection pointer generating module, used for tracking the coordinates of a first key point and a second key point of a user's hand in the display interface based on a preset algorithm to generate a moving selection pointer; The target component determination module is used to determine the extended screen component intersecting with the moving selection pointer as the target component when the user's hand is detected to be moving, so as to control the target component.

19. A computer device comprising a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the control method of the extended screen assembly as described in any one of claims 1 to 17 when executing the computer program.

20. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, enables the processor to implement the control method of the extended screen assembly according to any one of claims 1 to 17.

Images