Control method and device of extension screen component, computer equipment and storage medium

Abstract

The invention relates to the technical field of intelligent wearing, and particularly discloses a control method and device of an expansion screen assembly, computer equipment and a storage medium. Identifying a screen profile of the user device to activate a screen extension mode; based on the relative position relationship between each expansion screen component and the screen contour, displaying the expansion screen components and a screen of the user equipment in the display interface; tracking a coordinate of a first key point and a coordinate of a second key point of the hand of the user, and generating a mobile selection pointer; and when the hand of the user moves, determining the extension screen component intersected with the movement selection pointer as a target component so as to control the target component. According to the method, the movement selection pointer is obtained through the two key points of the hand of the user, then the intersection condition of the movement selection pointer and the expansion screen assembly is obtained when the user moves the hand, the expansion screen assembly intersecting with the movement selection pointer serves as the target assembly, user operation is simplified, and user experience is improved. And the portability and the accuracy of selecting the expansion screen component by the user are improved.

Description

Technical Field

[0001] The present application relates to the field of smart wearable technology, and in particular to a control method, device, computer equipment and storage medium for an extended screen assembly. Background Art

[0002] At present, although there are some extended screen products for smart glasses (such as AR (Augmented Reality) glasses) and user devices (such as mobile phones, tablets, etc.) on the market, most of the operations on the extended screen of the user device use gestures or voice, which requires users to spend time learning and adapting to the operation methods. Moreover, when the user's gestures are not clear enough and the language expression is not clear, the determined extended screen is not what the user needs, and the accuracy of the extended screen selection is low. Therefore, how to improve the portability and accuracy of users' selection of extended screens has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a control method, device, computer equipment and storage medium for an extended screen assembly to improve the portability of operation and the accuracy of extended screen selection.

[0004] In a first aspect, the present application provides a method for controlling an extended screen assembly, the method comprising:

[0005] Identify the screen outline of the user's device to activate the screen extension mode;

[0006] Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface;

[0007] Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a moving selection pointer;

[0008] When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0009] In a second aspect, the present application further provides a control device for an extended screen assembly, the device comprising:

[0010] A screen outline recognition module, used to recognize the screen outline of the user device to activate the screen extension mode;

[0011] A display interface display module, configured to display the extended screen components and the screen of the user device in a display interface based on the relative positional relationship between each extended screen component and the screen outline;

[0012] A moving selection pointer generating module, used for tracking the coordinates of a first key point and a second key point of a user's hand in the display interface based on a preset algorithm to generate a moving selection pointer;

[0013] The target component determination module is used to determine the extended screen component intersecting with the moving selection pointer as the target component when the user's hand is detected to be moving, so as to control the target component.

[0014] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the control method of the extended screen assembly as described above when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the control method of the extended screen assembly as described above.

[0016] The present application discloses a control method, device, computer equipment and storage medium for an extended screen component, which can identify the screen outline of a user device to activate the screen extension mode; based on the relative position relationship between each extended screen component and the screen outline, display the extended screen component and the screen of the user device in a display interface; based on a preset algorithm, track the coordinates of the first key point of the user's hand and the coordinates of the second key point in the display interface to generate a mobile selection pointer; when the user's hand is detected to be moving, determine the extended screen component that intersects with the mobile selection pointer as a target component to control the target component. The method obtains the mobile selection pointer through the two key points of the user's hand, and then obtains the intersection of the mobile selection pointer and the extended screen component when the user moves his hand, and uses the extended screen component that intersects with the mobile selection pointer as the target component, which simplifies the user operation and improves the portability and accuracy of the user's selection of the extended screen component. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a first embodiment of a control method for an extended screen assembly provided in an embodiment of the present application;

[0019] Figure 2It is a schematic diagram of a display interface of a control method of an extended screen assembly provided in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the effect of adaptive changes of components in a control method of an extended screen component provided in an embodiment of the present application;

[0021] Figure 4 is a schematic flow chart of a second embodiment of a control method for an extended screen assembly provided in an embodiment of the present application;

[0022] Figure 5 is a schematic flow chart of a third embodiment of a control method for an extended screen assembly provided in an embodiment of the present application;

[0023] Figure 6 A schematic block diagram of a control device for an extended screen assembly provided in an embodiment of the present application;

[0024] Figure 7 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0028] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] The embodiments of the present application provide a control method, device, computer equipment and storage medium for an extended screen component. The control method for the extended screen component can be applied to a server, and a moving selection pointer is generated by a hand key point to determine the target component, thereby improving the portability and accuracy of the user's selection of the extended screen component. The server can be an independent server or a server cluster.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] See also Figure 1 , Figure 1 This is a schematic flow chart of the first embodiment of a control method for an extended screen component provided by an embodiment of the present application. The control method for an extended screen component can be applied to a server to generate a moving selection pointer through a hand key point and then determine a target component, thereby improving the portability and accuracy of a user's selection of an extended screen component.

[0032] like Figure 1 As shown, the control method of the extended screen assembly specifically includes steps S101 to S1034.

[0033] S101: Identify a screen outline of a user device to activate a screen extension mode.

[0034] In one embodiment, when the screen of a user device (such as a mobile phone, tablet computer, etc.) is turned on by the user, the screen brightness will change.

[0035] In a specific embodiment, when the smart wearable device (such as AR glasses, AR helmets, etc.) detects that the screen of the user device is lit, it calls its own image acquisition module to identify the screen outline of the user device and activate the screen extension mode. It can be understood that the image acquisition module can be a camera built into the smart wearable device. In another embodiment, the image acquisition module can also be a camera connected to the smart wearable device.

[0036] In one embodiment, Figure 2 As shown, in the screen extension mode, the user device and the extended screen component are in a bound state, and the display interface presented to the user by the smart wearable device will display the screen of the user device and the extended screen component of the user device or the smart wearable device.

[0037] It can be understood that the extended screen component can be an application component in the user device, which can display the application content of the user device, such as a video component that can display the played video.

[0038] In another embodiment, the extended screen component may also be a functional component in the smart wearable device, which may display relevant functions of the smart wearable device for the user to query and operate.

[0039] Among them, the extended screen component is a high-performance cross-terminal development framework for smart large-screen devices, a fast application engine running on smart wearable devices, which can help developers quickly build application services for smart wearable devices and provide multi-screen interaction capabilities.

[0040] S102: Based on the relative position relationship between each extended screen component and the screen outline, display the extended screen components and the screen of the user device in a display interface.

[0041] Based on the relative position relationship between each extended screen component and the screen outline, before displaying the extended screen component and the screen of the user device in the display interface, it also includes: obtaining the outline coordinates of the screen outline, and based on the outline coordinates, obtaining the first position of the user device in the display interface; based on the preset size of each extended screen component, the screen size of the user device and the first position, obtaining the relative position relationship between each extended screen component and the user device.

[0042] In one embodiment, the smart wearable device calls the image acquisition module to identify the screen outline of the user device, and records the outline coordinates of the screen outline of the user device, and selects the coordinates of the upper left corner point and the lower right corner point of the user device screen from the outline coordinates to indicate the first position of the screen of the user device in the display interface. For example, after the user turns on the mobile phone screen, the AR glasses identify the outline of the mobile phone screen, obtain the coordinates of the upper left corner point and the lower right corner point of the mobile phone screen outline to indicate the first position of the mobile phone screen.

[0043] In one embodiment, a smart wearable device (such as smart glasses, etc.) projects the screen content of a user device or the related content of the smart wearable device itself as AR content in the form of an extended screen component and displays it in the external space of the screen outline, thereby forming an extended display of the user device screen.

[0044] In one embodiment, the display interface is an interface displayed by the smart wearable device to the user, and is captured by the image acquisition module of the smart wearable device. It is understandable that, when the image captured by the image acquisition module is not cropped, the image size captured by the image acquisition module is the same as the size of the display interface. The image size can be calculated based on the preset image resolution and pixel value of the image acquisition device. For example, if the pixels of a picture are 600×800 pixels and the resolution is 300×400, then the actual length and width are: 600 / 300=2 inches, 800 / 400=2 inches, and the length and width are both about 5 cm, that is, the actual picture is 5 cm long and 5 cm wide. The coordinate information of the display interface can be obtained based on the image size information collected by the image acquisition module, and the coordinate system can be determined based on the image size information collected by the image acquisition module. For example, when the image acquisition module is a camera device, it is known that the width of the image captured by the camera is c. w , height c h , taking the upper left corner as the origin of the coordinate system (0, 0), the leftmost side of the display interface as the ordinate axis of the coordinate system, and the top of the display interface as the abscissa axis, the coordinates of the lower right corner of the display interface can be obtained as (v w , c h ).

[0045] In one embodiment, the relative position relationship between the extended screen components and the first position of the user device is determined according to the size of each extended screen component and the screen size of the user device. For example, the user device is a mobile phone, the screen size of the mobile phone is 7.5 cm wide and 13.5 cm long, and the sizes of the extended screen components are: first extended screen component: 2 cm × 2 cm, second component: 1.5 cm × 1.5 cm, third component: 1.5 cm × 2 cm, etc., then the extended screen components of appropriate sizes can be arranged in rows at the top of the mobile phone and in columns on the side of the mobile phone according to the width and length of the mobile phone screen, and the relative position relationship between each extended screen component and the mobile phone screen is obtained, for example, the first extended screen component is located in the first column and second row on the left side of the mobile phone screen.

[0046] In one embodiment, the position coordinates of each extended screen component are determined based on the relative position relationship between each extended screen component and the first position of the user device and the size of the extended screen component. For example, the upper left corner point of the user device is (20,30), and the lower right corner point is (30,50), then the first position is expressed as ((20,30), (30,50)), the size of the first extended screen component is 2 cm wide and 2 cm long, and the relative position relationship with the user device is that the first extended screen component is located in the first row and first column at the top of the user device, then the coordinates of the upper left corner point of the first extended screen component can be obtained as (20,28), and the lower right corner point is (22,32), and the position coordinates of the first extended screen component can be expressed as

[0047] T1 (20,28)，(22,32)

[0048] The screen size of the user device is known and can be obtained by measurement or by querying the product information of the user device. The size of each extended screen component can be set by the user or preset by the developer.

[0049] The positions of the various extended screen components are shown as follows:

[0050]

[0051] Among them, x1y1 represents the coordinates of the upper left corner of each extended screen component, x2y2 represents the coordinates of the lower right corner of each extended screen component, and T1, T2...Tn are used to represent the position points of the 1st, 2nd...nth extended screen components. That is, the coordinates of the upper left corner and the lower right corner of the extended screen component are used to represent the position of the extended screen component.

[0052] In one embodiment, according to the obtained positions of the respective extended screen components in the display interface and the first position of the screen outline of the user device in the display interface, the respective extended screen components are displayed around the screen of the user device, for example, Figure 2 As shown, the various extended screen components are distributed on the left and right sides and the top of the user device screen.

[0053] S103: Based on a preset algorithm, track the coordinates of a first key point and a second key point of the user's hand in the display interface to generate a moving selection pointer.

[0054] Based on a preset algorithm, the coordinates of the first key point and the coordinates of the second key point of the user's hand in the display interface are tracked to generate a moving selection pointer, including: based on the preset algorithm, the user's hand is identified to determine the first key point and the second key point; when the user's hand is detected to be moving, the first key point coordinates of the first key point and the second key point coordinates of the second key point in the display interface are tracked; and a line segment passing through the first key point coordinates and the second key point coordinates is obtained as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0055] In one embodiment, the preset algorithm is a hand key point algorithm, which can track the coordinates of the hand key points in real time according to the movement of the user's hand. For example, a hand key point detection method based on MediaPipe (MediaPipe is a framework for building cross-platform machine learning solutions) includes a palm detector and a hand coordinate model. The palm detector is used to determine the position of the user's hand, and the hand coordinate model is used to obtain the coordinates of the hand key points. When a human hand is detected, the coordinates of the human hand key points can be tracked in real time during the movement of the human hand.

[0056] In a specific embodiment, a preset algorithm is stored in the smart wearable device. When in the screen expansion mode, the smart wearable device calls the preset algorithm to identify 21 key points of the hand, and determines the first key point and the second key point among the key points. For example, the first joint of the user's thumb is used as the first key point, and the fingertip of the user's thumb is used as the second key point.

[0057] In one embodiment, the movement of the user's hand may cause the first key point coordinates and the second key point coordinates to change. When the changes in the first key point coordinates and the second key point coordinates are detected, it can be determined that the user's hand is moving.

[0058] In one embodiment, when the user's hand moves, the first key point coordinates corresponding to the first key point of the user's hand and the second key point coordinates corresponding to the second key point are tracked in real time through a preset algorithm, and the line segment passing through the first key point coordinates and the second key point coordinates is used as a moving selection pointer.

[0059] In one embodiment, the direction of the moving selection pointer points from the first key point to the second key point. For example, a line segment starting from the first joint of the thumb and connecting the tip of the thumb is the moving selection pointer, and the direction of the first joint of the thumb pointing to the fingertip is the direction of the moving selection pointer.

[0060] S104: When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0061] When the movement of the user's hand is detected, after the extended screen component intersecting with the mobile selection pointer is determined as the target component, it also includes: identifying the operation action of the user's hand, and based on the operation action, displaying the corresponding interface of the target component to the user.

[0062] Based on the operation action, the corresponding interface of the target component is displayed to the user, including: identifying the operation action, comparing the operation action with the gesture actions in a preset gesture library, and determining the operation instruction corresponding to the operation action; when the operation instruction is to display the details page, the details page of the target component is displayed; when the operation instruction is to switch components, the target component is switched based on a preset component switching order; when the operation instruction is to return to the display interface, exit the details page and return to the display interface.

[0063] In one embodiment, when the user moves his hand, the mobile selection pointer will change accordingly. For example, when the user changes the direction of his thumb, the direction of the mobile selection pointer generated by the user's thumb will change accordingly. During the movement, the mobile selection pointer will intersect with the extended screen components around the user device screen in the display interface, and the extended screen components intersecting with the mobile selection pointer will be used as the target components.

[0064] In one embodiment, after the target component is determined, the corresponding display instruction is generated according to the preset component selection display effect, and the target component is adaptively changed, for example, the size of the component is changed, the transparency is changed, such as Figure 3 As shown, after the target component is selected, the target component will become larger to achieve a similar floating effect. At the same time, the transparency of other unselected components will increase to highlight the target component.

[0065] In one embodiment, after determining the target component, the operation action of the user's hand is identified, and the operation action of the user's hand is compared with the gesture action in the preset gesture library. The gesture library includes gesture actions and operation instructions corresponding to the gesture actions. After determining the gesture action corresponding to the operation action, the corresponding operation instruction can be obtained. The operation instruction includes displaying the detail page, switching components, returning to the main interface (i.e., display interface), etc. Among them, the switching component can be switched according to the preset component switching order.

[0066] The above embodiments provide a control method, device, computer equipment and storage medium for an extended screen component, which identifies the screen outline of a user device to activate the screen extension mode; based on the relative position relationship between each extended screen component and the screen outline, displays the extended screen component and the screen of the user device in a display interface; based on a preset algorithm, tracks the coordinates of the first key point of the user's hand and the coordinates of the second key point in the display interface to generate a mobile selection pointer; when the user's hand is detected to be moving, determines the extended screen component that intersects with the mobile selection pointer as a target component to control the target component. This method obtains the mobile selection pointer through the two key points of the user's hand, and then obtains the intersection of the mobile selection pointer and the extended screen component when the user moves his hand, and uses the extended screen component that intersects with the mobile selection pointer as the target component, which simplifies the user operation and improves the portability and accuracy of the user's selection of the extended screen component.

[0067] See also Figure 4 , Figure 4 This is a schematic flow chart of the second embodiment of a control method for an extended screen component provided by an embodiment of the present application. The control method for an extended screen component can be applied to a server to generate a moving selection pointer through a hand key point and then determine a target component, thereby improving the portability and accuracy of a user's selection of an extended screen component.

[0068] like Figure 4 As shown, step S104 of the control method of the extended screen assembly specifically includes steps S201 to S203.

[0069] S201, when the user's hand is detected to be moving, determining a target area in the display interface based on a comparison result of a current slope of the moving selection pointer and a preset slope;

[0070] S202, taking the extended screen component in the target area as a component to be determined;

[0071] S203: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, determine that the component to be determined intersects with the moving selection pointer, and use the component to be determined as the target component.

[0072] Based on the comparison result of the current slope of the mobile selection pointer and the preset slope, before determining the target area in the display interface, it also includes: obtaining the current slope of the mobile selection pointer based on the first key point coordinates and the second key point coordinates; obtaining the coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module; determining the upper left corner point and the lower right corner point of the display interface based on the coordinate information, and obtaining the slope of the line segment passing through the upper left corner point and the lower right corner point as the preset slope.

[0073] In a specific embodiment, the display interface is an interface displayed by the smart wearable device to the user, and is captured by the image acquisition module of the smart wearable device. It can be understood that, when the image captured by the image acquisition module is not cropped, the size of the image captured by the image acquisition module is the same as the size of the display interface. The size information of the display interface can be obtained based on the image size information collected by the image acquisition module, and the coordinate system can be determined based on the image size information collected by the image acquisition module. For example, when the image acquisition module is a camera device, it is known that the width of the image captured by the camera is c w , height c h , taking the upper left corner as the origin of the coordinate system (0, 0), the leftmost side of the display interface as the ordinate axis of the coordinate system, and the top of the display interface as the abscissa axis, the coordinates of the lower right corner of the display interface can be obtained as (c w , c h ), the slope of the line segment formed by the upper left corner point and the lower right corner point is c k , c k as the preset slope.

[0074] In a specific embodiment, the coordinates of the first key point are set to t1, and the coordinates of the second key point are set to t2. The current slope t of the moving selection pointer can be obtained based on the two coordinate points t1 and t2. k .

[0075] In one embodiment, the current slope t of the moving selection pointer is k With preset slope c k By comparison, the area where the target component is located can be preliminarily determined and the area is taken as the target area. All extended screen components in the target area are taken as components to be determined whether they intersect with the extension line of the moving selection pointer.

[0076] In a specific embodiment, if t k <c k, then extend the line segment t2t1 to the leftmost position x=0 of the display interface, obtain the intersection point x0 with the leftmost edge line of the display interface, and obtain the line segment passing through t2, t1, x0. That is, the extended line of the moving selection pointer will pass through the left area of ​​the user device and intersect with the preset edge of a certain extended screen component in the left area of ​​the user device. At this time, the left area is taken as the target area. Figure 3 As shown in the first figure, at this time, the current slope t of the moving selection pointer k Less than the slope of the display interface (i.e. the preset slope) c k , the extension line of the moving selection pointer intersects with one of the extended screen components in the left area of ​​the user device screen in the display interface, then the left area is taken as the target area, and all the extended screen components in the left area are taken as components to be judged.

[0077] In one embodiment, if t k >c k , then extend the line segment t2t1 to the top edge of the display interface at y=0, obtain the intersection y0 with the top edge of the display interface, and obtain the line segment passing through t2, t1, y0. That is, the extension line of the moving selection pointer will pass through the top area of ​​the user device and intersect with the preset edge of a certain extended screen component in the top area of ​​the user device. At this time, the top area is taken as the target area. Figure 3 As shown in the third figure, at this time, the current slope t of the moving selection pointer k Greater than the slope of the display interface (i.e. the preset slope) c k , the extension line of the moving selection pointer intersects with one of the extended screen components in the top area of ​​the user device screen in the display interface, then the top area is taken as the target area, and all the extended screen components in the top area are taken as components to be judged.

[0078] In one embodiment, if t k =c k , then extend the line segment t2t1 to the upper left corner origin (0,0), obtain the intersection t0 with the upper left corner of the display interface, and obtain the line segment passing through t2, t1, t0. That is, the extended line of the moving selection pointer will pass through the upper left corner area of ​​the user device and intersect with the preset edge of a certain extended screen component in the upper left corner area of ​​the user device. At this time, the upper left corner area is used as the target area. Figure 3 As shown in the second figure, at this time, the current slope t of the moving selection pointer k Equal to the slope of the display interface (i.e. the preset slope) c k , the extension line of the moving selection pointer will intersect with the extended screen component existing in the upper left corner area of ​​the user device screen in the display interface, then the upper left corner area is used as the target area, and the extended screen component in the upper left corner area is used as the component to be judged.

[0079] In one embodiment, when the user uses his left hand to operate, the finger may point to the right area of ​​the user device, and the extension line of the moving selection pointer has a chance to intersect with the preset edge of the extended screen assembly in the right area.

[0080] In one embodiment, the side of the component to be judged close to the screen of the user device is taken as the preset side. For example, when the left area of ​​the user device is determined as the target area, the extended screen component in the left area is taken as the component to be judged, and the right border of the component to be judged is taken as the preset side; when the right area of ​​the user device is determined as the target area, the extended screen component in the right area is taken as the component to be judged, and the left border of the component to be judged is taken as the preset side; when the top area of ​​the user device is determined as the target area, the extended screen component in the top area is taken as the component to be judged, and the bottom border of the component to be judged is taken as the preset side.

[0081] When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the moving selection pointer, and the component to be determined is used as the target component.

[0082] It can be understood that when determining the target component, firstly, the target area where the target component may be located is determined according to the relationship between the slope of the moving selection pointer and the preset slope, and all the extended screen components in the target area are taken as components to be determined whether they are the target components, and then the edge of the component to be determined adjacent to the user device screen is taken as the preset edge, and it is determined whether the extension line of the moving selection pointer intersects with the preset edge, so that only one target component can be determined. For example, Figure 3 As shown in the second figure in , the slope of the moving selection pointer is equal to the preset slope. At this time, the upper left corner area of ​​the user device screen is taken as the target area, and the component in the upper left corner is taken as the component to be determined, and the target component is determined in the components to be determined. Even if the moving selection pointer also intersects with the extended screen component in the left area, since the left area is not the target area, the extended screen component in the left area has been excluded, so the moving selection pointer only intersects with the extended screen component in the upper left corner.

[0083] The above-mentioned embodiment provides a control method, device, computer equipment and storage medium for an extended screen component. When the user's hand is detected to be moving, based on the comparison result of the current slope of the mobile selection pointer and the preset slope, a target area is determined in the display interface; the extended screen component in the target area is used as a component to be determined; when the extension line of the mobile selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the mobile selection pointer, and the component to be determined is used as the target component. This method determines the target slope by comparing the current slope of the mobile selection pointer with the preset slope, obtains the extended screen component in the target area as the component to be determined, and can accurately determine the extended screen component that intersects with the mobile selection pointer as the target component by determining whether the preset edge of the component to be determined intersects with the extension line corresponding to the mobile selection pointer, thereby improving the accuracy of the determination of the target component.

[0084] See also Figure 5 , Figure 5 This is a schematic flow chart of the third embodiment of a control method for an extended screen component provided by an embodiment of the present application. The control method for an extended screen component can be applied to a server to generate a moving selection pointer through a hand key point and then determine a target component, thereby improving the portability and accuracy of a user's selection of an extended screen component.

[0085] like Figure 5 As shown, the steps of the control method of the extended screen component include determining when the extension line of the moving selection pointer intersects with the preset edge of the component to be judged, and before determining that the component to be judged intersects with the moving selection pointer, and specifically also include steps S301 to S304.

[0086] S301, obtaining the coordinates of two endpoints of the preset edge of the component to be determined;

[0087] S302, obtaining a line segment expression corresponding to the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0088] S303, obtaining a first value and a second value based on the two endpoint coordinates and the line segment expression respectively;

[0089] S304: When the first value is a negative number and the second value is a positive number, or when the first value is a positive number and the second value is a negative number, determine that the extension line of the moving selection pointer intersects with a preset edge of the component to be determined.

[0090] In a specific embodiment, the line segment expression of the current moving selection pointer can be obtained according to the first key point coordinate t1, the second key point coordinate t2, and the intersection coordinate x0 or y0 or t0 of the extension line of the moving selection pointer and the horizontal / vertical coordinate axis. For example, the line segment expression is y=ax+b, and the values ​​of a and b can be obtained by substituting the first key point coordinate t1, the second key point coordinate t2, and the intersection coordinate x0 or y0 or t0 to obtain the line segment expression corresponding to the current moving selection pointer.

[0091] In a specific embodiment, the endpoint coordinates of the two endpoints of the preset edge are obtained, and the two endpoint coordinates are respectively substituted into the expression w=y-ax-b to obtain a first value w1 and a second value w2. The values ​​of w1 and w2 are compared. When w1 and w2 are both positive or negative, it indicates that the two endpoints of the preset edge are on the same side of the moving selection pointer, that is, the preset edge of the current component to be judged does not intersect with the extension line of the current moving selection pointer; when w1 is positive and w2 is negative or w1 is negative and w2 is positive, it indicates that the two endpoints of the preset edge are on both sides of the moving selection pointer, that is, the preset edge of the current component to be judged intersects with the extension line of the current moving selection pointer, and the component to be judged is determined to be the target component. Among them, the expression w=y-ax-b is used to determine whether a certain point is on the same side of the line segment y=ax+b, and the values ​​of a and b are the values ​​in the current moving selection pointer expression. Substituting the coordinates (x, y) of a certain point into the expression to obtain the value of w can determine whether the point (x, y) is on the same side of the current line segment y=ax+b. For example, the line segment expression of the current moving selection pointer is y=2x+1, then w=y-2x-1 is obtained, and the endpoint coordinates of the preset edge are (2,1) and (1,1). Substituting into w=y-2x-1, we get w1=-1, w2=-2. At this time, w1 and w2 are both negative numbers, indicating that the endpoints of the preset edge are on the same side of the moving selection pointer, that is, the preset edge will not intersect with the extension line of the moving selection pointer.

[0092] The above-mentioned embodiment provides a control method, device, computer equipment and storage medium for an extended screen component, which obtains the coordinates of the two endpoints of the preset edge of the component to be judged; based on the first key point coordinates and the second key point coordinates, obtains the line segment expression corresponding to the moving selection pointer; based on the two endpoint coordinates and the line segment expression, obtains the first value and the second value respectively; when the first value is negative and the second value is positive, or the first value is positive and the second value is negative, determines that the extension line of the moving selection pointer intersects with the preset edge of the component to be judged. This method obtains the endpoint coordinates of the preset edge of the component to be judged and the line segment expression corresponding to the moving selection pointer, and by determining whether the two endpoints of the preset edge of the component to be judged are on the same side of the moving selection pointer, it can accurately determine whether the preset edge of the component to be judged intersects with the extension line of the moving selection pointer, thereby improving the accuracy of determining the target component.

[0093] See also Figure 6 , Figure 6 The embodiment of the present application provides a schematic block diagram of a control device for an extended screen assembly, wherein the control device for the extended screen assembly is used to execute the aforementioned control method for the extended screen assembly. The control device for the extended screen assembly can be configured on a server.

[0094] like Figure 6 As shown, the control device 400 of the extended screen assembly includes:

[0095] A screen outline recognition module 401 is used to recognize the screen outline of the user device to activate the screen extension mode;

[0096] A display interface display module 402, configured to display the extended screen components and the screen of the user device in a display interface based on the relative position relationship between each extended screen component and the screen outline;

[0097] A moving selection pointer generating module 403 is used to track the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm to generate a moving selection pointer;

[0098] The target component determination module 404 is used to determine the extended screen component intersecting with the moving selection pointer as a target component when the user's hand is detected to be moving, so as to control the target component.

[0099] In one embodiment, the mobile selection pointer generation module 403 includes:

[0100] A key point determination unit, configured to identify the user's hand based on the preset algorithm and determine a first key point and a second key point;

[0101] a coordinate tracking unit, configured to track a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface when detecting movement of the user's hand;

[0102] A moving selection pointer determining unit is used to obtain a line segment passing through the coordinates of the first key point and the coordinates of the second key point as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0103] In one embodiment, the target component determination module 404 includes:

[0104] a target area determination unit, configured to determine a target area in the display interface based on a comparison result between a current slope of the moving selection pointer and a preset slope when the user's hand is detected to be moving;

[0105] A component-to-be-judged determining unit, configured to take the extended screen component in the target area as a component to be judged;

[0106] A target component determination unit is used to determine that the component to be determined intersects with the moving selection pointer when the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, and to use the component to be determined as the target component.

[0107] In one embodiment, the target component determination module further includes a slope acquisition submodule, and the slope acquisition submodule includes:

[0108] a current slope obtaining unit, configured to obtain a current slope of the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0109] A coordinate information obtaining unit, used to obtain the coordinate information of the display interface based on the size information of the display interface acquired by the image acquisition module;

[0110] A preset slope obtaining unit is used to determine the upper left corner point and the lower right corner point of the display interface based on the coordinate information, and obtain the slope of the line segment passing through the upper left corner point and the lower right corner point as the preset slope.

[0111] In one embodiment, the target component determination module further includes an intersection determination submodule, and the intersection determination submodule includes:

[0112] An endpoint acquisition unit, used to acquire the coordinates of two endpoints of a preset edge of the component to be determined;

[0113] A line segment expression obtaining unit, used for obtaining a line segment expression corresponding to the moving selection pointer based on the first key point coordinates and the second key point coordinates;

[0114] A value obtaining unit, used to obtain a first value and a second value respectively based on the two endpoint coordinates and the line segment expression;

[0115] An intersection determination unit is used to determine that the extension line of the moving selection pointer intersects with the preset edge of the component to be judged when the first value is negative and the second value is positive, or when the first value is positive and the second value is negative.

[0116] In one embodiment, the control device 400 of the extended screen assembly further includes a position relationship acquisition module, and the position relationship acquisition module includes:

[0117] A first position obtaining unit, configured to obtain the outline coordinates of the screen outline, and obtain a first position of the user equipment in the display interface based on the outline coordinates;

[0118] A relative position relationship obtaining unit is used to obtain the relative position relationship between each of the extended screen components and the user equipment based on the preset size of each of the extended screen components, the screen size of the user equipment and the first position.

[0119] In one embodiment, the control device 400 of the extended screen assembly further includes a component display module, and the component display module includes:

[0120] The corresponding page display unit is used to identify the operation action of the user's hand and display the corresponding interface of the target component to the user based on the operation action.

[0121] In one embodiment, the corresponding page display unit includes:

[0122] An operation instruction determination subunit, used to identify the operation action, compare the operation action with gesture actions in a preset gesture library, and determine the operation instruction corresponding to the operation action;

[0123] A details page display subunit, used to display the details page of the target component when the operation instruction is to display the details page;

[0124] A component switching subunit, configured to switch the target component based on a preset component switching sequence when the operation instruction is a component switching;

[0125] The display interface returning subunit is used to exit the details page and return to the display interface when the operation instruction is to return to the display interface.

[0126] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0127] The above-mentioned device can be implemented in the form of a computer program. Figure 7 Runs on the computer device shown.

[0128] See also Figure 7 , Figure 7 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0129] See also Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0130] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any control method of the extended screen assembly.

[0131] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0132] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any control method of the extended screen component.

[0133] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0135] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0136] Identify the screen outline of the user's device to activate the screen extension mode;

[0137] Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface;

[0138] Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a moving selection pointer;

[0139] When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

[0140] In one embodiment, when the processor tracks the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm and generates a moving selection pointer, it is used to implement:

[0141] Based on the preset algorithm, the user's hand is identified to determine a first key point and a second key point;

[0142] When detecting movement of the user's hand, tracking a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface;

[0143] A line segment passing through the coordinates of the first key point and the coordinates of the second key point is obtained as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

[0144] In one embodiment, when the processor determines the extended screen component intersecting with the moving selection pointer as the target component when detecting the movement of the user's hand, it is configured to implement:

[0145] When the user's hand is detected to be moving, based on a comparison result between a current slope of the moving selection pointer and a preset slope, a target area is determined in the display interface;

[0146] Taking the extended screen component in the target area as the component to be determined;

[0147] When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the moving selection pointer, and the component to be determined is used as the target component.

[0148] In one embodiment, before determining the target area in the display interface based on the comparison result between the current slope of the moving selection pointer and the preset slope, the processor is further configured to implement:

[0149] Based on the first key point coordinates and the second key point coordinates, obtaining a current slope of the moving selection pointer;

[0150] Based on the size information of the display interface acquired by the image acquisition module, obtaining the coordinate information of the display interface;

[0151] Based on the coordinate information, the upper left corner point and the lower right corner point of the display interface are determined, and the slope of the line segment passing through the upper left corner point and the lower right corner point is obtained as the preset slope.

[0152] In one embodiment, when the extended line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer, the processor is further configured to implement:

[0153] Obtain the coordinates of two endpoints of the preset edge of the component to be determined;

[0154] Based on the first key point coordinates and the second key point coordinates, obtaining a line segment expression corresponding to the moving selection pointer;

[0155] Based on the two endpoint coordinates and the line segment expression, respectively obtain a first value and a second value;

[0156] When the first value is a negative number and the second value is a positive number, or when the first value is a positive number and the second value is a negative number, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be determined.

[0157] In one embodiment, the processor is further used to implement, before displaying the extended screen components and the screen of the user device in the display interface based on the relative position relationship between each extended screen component and the screen outline:

[0158] Acquire contour coordinates of the screen contour, and obtain a first position of the user equipment in the display interface based on the contour coordinates;

[0159] Based on the preset size of each of the extended screen components, the screen size of the user equipment and the first position, the relative position relationship between each of the extended screen components and the user equipment is obtained.

[0160] In one embodiment, after the processor determines the extended screen component intersecting with the moving selection pointer as the target component when detecting the movement of the user's hand, it is further configured to implement:

[0161] Identify the operation action of the user's hand, and based on the operation action, display the corresponding interface of the target component to the user.

[0162] In one embodiment, when the processor displays the corresponding interface of the target component to the user based on the operation action, it is used to implement:

[0163] Identify the operation action, compare the operation action with gesture actions in a preset gesture library, and determine the operation instruction corresponding to the operation action;

[0164] When the operation instruction is to display a detail page, display the detail page of the target component;

[0165] When the operation instruction is a component switching, switching the target component based on a preset component switching sequence;

[0166] When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

[0167] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any control method of an extended screen assembly provided in an embodiment of the present application.

[0168] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0169] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A control method for an extended screen assembly, characterized in that: include: Identify the screen outline of the user's device to activate the screen extension mode; Based on the relative positional relationship between each extended screen component and the screen outline, displaying the extended screen component and the screen of the user device in a display interface; Based on a preset algorithm, the coordinates of a first key point and a second key point of the user's hand in the display interface are tracked to generate a moving selection pointer; When the movement of the user's hand is detected, the extended screen component intersecting with the moving selection pointer is determined as a target component to control the target component.

2. The control method of the extended screen assembly according to claim 1, characterized in that: The step of tracking the coordinates of the first key point and the second key point of the user's hand in the display interface based on a preset algorithm to generate a moving selection pointer includes: Based on the preset algorithm, the user's hand is identified to determine a first key point and a second key point; When detecting that the user's hand moves, tracking a first key point coordinate of the first key point and a second key point coordinate of the second key point in the display interface; A line segment passing through the coordinates of the first key point and the coordinates of the second key point is obtained as the moving selection pointer, wherein the direction of the moving selection pointer is from the first key point to the second key point.

3. The control method of the extended screen assembly according to claim 1, characterized in that: The step of determining, when detecting that the user's hand is moving, an extended screen component intersecting with the moving selection pointer as a target component comprises: When the user's hand is detected to be moving, based on a comparison result between a current slope of the moving selection pointer and a preset slope, a target area is determined in the display interface; Taking the extended screen component in the target area as the component to be determined; When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, it is determined that the component to be determined intersects with the moving selection pointer, and the component to be determined is used as the target component.

4. The control method of the extended screen assembly according to claim 3, characterized in that: The method further includes, before determining the target area in the display interface based on the comparison result of the current slope of the moving selection pointer and the preset slope: Based on the first key point coordinates and the second key point coordinates, obtaining a current slope of the moving selection pointer; Based on the size information of the display interface acquired by the image acquisition module, obtaining the coordinate information of the display interface; Based on the coordinate information, the upper left corner point and the lower right corner point of the display interface are determined, and the slope of the line segment passing through the upper left corner point and the lower right corner point is obtained as the preset slope.

5. The control method of the extended screen assembly according to claim 3, characterized in that: When the extension line of the moving selection pointer intersects with the preset edge of the component to be determined, before determining that the component to be determined intersects with the moving selection pointer, the method further includes: Obtain the coordinates of two endpoints of the preset edge of the component to be determined; Based on the first key point coordinates and the second key point coordinates, obtaining a line segment expression corresponding to the moving selection pointer; Based on the two endpoint coordinates and the line segment expression, respectively obtain a first value and a second value; When the first value is a negative number and the second value is a positive number, or when the first value is a positive number and the second value is a negative number, it is determined that the extension line of the moving selection pointer intersects with the preset edge of the component to be determined.

6. The control method of the extended screen assembly according to any one of claims 1 to 5, characterized in that: Based on the relative positional relationship between each extended screen component and the screen outline, before displaying the extended screen component and the screen of the user equipment in the display interface, the method further includes: Acquire contour coordinates of the screen contour, and obtain a first position of the user equipment in the display interface based on the contour coordinates; Based on the preset size of each of the extended screen components, the screen size of the user equipment and the first position, the relative position relationship between each of the extended screen components and the user equipment is obtained.

7. The control method of the extended screen assembly according to claim 1, characterized in that: After the extended screen component intersecting with the moving selection pointer is determined as a target component when the user's hand is detected to be moving, the method further includes: Identify the operation action of the user's hand, and based on the operation action, display the corresponding interface of the target component to the user.

8. The control method of the extended screen assembly according to claim 7, characterized in that: The displaying the corresponding interface of the target component to the user based on the operation action includes: Identify the operation action, compare the operation action with gesture actions in a preset gesture library, and determine the operation instruction corresponding to the operation action; When the operation instruction is to display a detail page, display the detail page of the target component; When the operation instruction is a component switching, switching the target component based on a preset component switching sequence; When the operation instruction is to return to the display interface, exit the details page and return to the display interface.

9. A control device for an extended screen assembly, characterized in that: include: A screen outline recognition module, used to recognize the screen outline of the user device to activate the screen extension mode; A display interface display module, configured to display the extended screen components and the screen of the user device in a display interface based on the relative positional relationship between each extended screen component and the screen outline; A moving selection pointer generating module, used for tracking the coordinates of a first key point and a second key point of a user's hand in the display interface based on a preset algorithm to generate a moving selection pointer; The target component determination module is used to determine the extended screen component intersecting with the moving selection pointer as the target component when the user's hand is detected to be moving, so as to control the target component.

10. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the control method of the extended screen assembly as described in any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the control method of the extended screen assembly according to any one of claims 1 to 8.

Images