Human-computer interaction control method and device, electronic equipment and storage medium

Abstract

The invention discloses a man-machine interaction control method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target user image when an interaction request is received, determining a human body interaction point of a target user from the target user image, and obtaining a human body interaction point of the target user; determining a self-adaptive interaction area which contains the human body interaction points and is matched with the body type of the target user from the target user image, and determining mapping points of the human body interaction points in the front-end interaction page according to the relative position relation between the human body interaction points and the self-adaptive interaction area; according to the technical scheme, distance interaction control over the front-end interaction page is achieved based on the mapping points, in addition, by determining the self-adaptive interaction area matched with the body type of the target user, the problems that the response stability of different users is insufficient due to large body type difference and the different users are not matched with interaction intentions of the users can be solved, and the user experience is improved. And the interaction accuracy and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a human-computer interaction control method, device, electronic device and storage medium. Background Art

[0002] With the development of information technology and computer science, human-computer interaction has become an indispensable part of modern technological products. Currently, when implementing control based on human-computer interaction, users usually operate on the device screen to achieve corresponding control, or the user controls through peripheral hardware, such as a mouse and keyboard. The interaction method is relatively simple. With the continuous upgrading of technology, users have an increasing demand for air-to-air interaction with devices. Traditional human-computer interaction control methods no longer meet user needs. There is an urgent need for a human-computer interaction control method that can achieve air-to-air interaction. Summary of the Invention

[0003] Based on this, a human-computer interaction control method, device, electronic device and storage medium are provided to solve the above technical problems.

[0004] In a first aspect, a human-computer interaction control method is provided, comprising: Upon receiving an interaction request, obtaining a target user image; determining a human body interaction point of a target user from the target user image, and determining an adaptive interaction area from the target user image that includes the human body interaction point and is adapted to the body shape of the target user; Determining a mapping point of the human body interaction point in the front-end interaction page according to a relative positional relationship between the human body interaction point and the adaptive interaction area; Interactive control of the front-end interactive page is achieved based on the mapping point.

[0005] In the above method, the mapping point of the human body interaction point in the front-end interaction page is determined according to the relative position relationship between the human body interaction point and the adaptive interaction area in the target user image, and the air interaction control of the front-end interaction page is realized based on the mapping point. In addition, by determining the adaptive interaction area that matches the body shape of the target user, the problem of insufficient response stability due to large differences in body shape between different users and the mismatch with the user's interaction intention can be avoided, thereby improving the interaction accuracy and user experience.

[0006] In one embodiment, determining a target user's body interaction point from the target user image, and determining an adaptive interaction area from the target user image that includes the body interaction point and is adapted to the target user's body shape, includes: Determining first position information of a key point of a human body of the target user in the target user image; An adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user is determined from the target user image according to the first position information.

[0007] For users of different body shapes, the first position information of the identified human body key points is different. The above method fully considers the impact of different body shape characteristics on the adaptive interaction area. The adaptive interaction area is determined by the first position information of the human body key points in the target user image, which improves the accuracy of interactive control and user experience satisfaction.

[0008] In one embodiment, determining, from the target user image according to the first position information, an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user includes: determining, from the target user image according to the first position information, a plurality of first candidate interaction areas that are adapted to the body shape of the target user; Acquire second position information of the human body interaction point in the target user image; An adaptive interaction area to which the human body interaction point belongs is determined from the multiple first candidate interaction areas according to the second position information.

[0009] In the above method, an adaptive interaction area is determined from multiple first candidate interaction areas according to the position of the human body interaction point, so that the determined adaptive interaction area matches the user's current interaction posture, thereby improving user experience satisfaction.

[0010] In one embodiment, the human body key points include a first key point, and the multiple first candidate interaction areas include a top interaction area located between the first key point and an upper interaction boundary line, and a lower interaction area located between the first key point and a lower interaction boundary line.

[0011] Since the positions of the first key points corresponding to users of different heights are different, the position of the first candidate interaction area obtained by the above method is related to the user's height, which can meet the requirements of users of different heights for the interaction area and improve the adaptability of the first candidate interaction area to the user's height. In addition, since each first candidate interaction area is essentially a sub-area of the entire area between the upper interaction boundary and the lower interaction boundary line, the determined adaptive interaction area is also a sub-area of the entire area between the upper interaction boundary and the lower interaction boundary line. Compared with directly using the entire area between the upper interaction boundary and the lower interaction boundary line as the adaptive interaction area, the area of the adaptive interaction area is reduced. Since the relative position relationship between the human body interaction point and the adaptive interaction area corresponds to the relative position relationship between the mapping point and the front-end interaction page, the above method reduces the area of the adaptive interaction area so that the user can trigger the corresponding control on the front-end interaction page by making small movements in the adaptive interaction area, which is more convenient for user interaction operations and improves the user experience.

[0012] In one embodiment, the human body key points also include a second key point located below the first key point, and the lower interaction area includes an intermediate interaction area between the first key point and the second key point and a bottom interaction area between the second key point and the interaction lower boundary line; the adaptive interaction area is one of the top interaction area, the intermediate interaction area and the bottom interaction area.

[0013] In the above method, the interactive area below is divided more finely through the second key point, and the area of the adaptive interactive area determined subsequently is further reduced, so that the user can trigger corresponding controls on the front-end interactive page by making small movements in the adaptive interactive area, further improving the user experience.

[0014] In one embodiment, the method for determining the upper boundary line of interaction includes: Determining a target detection frame covering the target user from the target user image; and determining an upper detection frame line of the target detection frame as the upper interaction boundary line; or, The human body key points include a third key point; obtaining a first height characteristic parameter of the target user; and determining the upper interaction boundary line according to the first height characteristic parameter and the first position information of the third key point.

[0015] Taking into account that the upper interaction boundary lines of users of different heights are different, in the above method, the upper interaction boundary line is determined by determining the target detection frame, or the upper interaction boundary line is determined by the first height feature parameter of the target user, so that the determined upper interaction boundary line is adapted to the user's height, further improving the adaptability of the first candidate interaction area to the user's height.

[0016] In one embodiment, the human body key points include a fourth key point, and the method for determining the interaction lower boundary line includes: Obtaining a second height characteristic parameter of the target user; The interaction lower boundary line is determined according to the second height characteristic parameter and the first position information of the fourth key point.

[0017] In the above method, the interaction lower boundary line is determined by the second height characteristic parameter of the target user, so that the determined interaction lower boundary line is adapted to the user's height, further improving the adaptability of the first candidate interaction area to the user's height.

[0018] In one embodiment, each of the first candidate interaction areas is located between two longitudinal boundary lines, and the longitudinal boundary lines are determined in a manner including: Determining a target detection frame covering the target user from the target user image; A vertical detection frame line of the target detection frame is determined as the vertical boundary line.

[0019] In the above method, the longitudinal boundary line is determined by determining the target detection frame, so that the determined longitudinal boundary line is adapted to the user's body shape, further improving the adaptability of the first candidate interaction area to the user's body shape.

[0020] In one embodiment, determining, from the target user image, an adaptive interaction area that includes the human body interaction point and is adapted to the target user's body shape includes: Determining a target detection frame covering the human interaction point and the target user from the target user image; An adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user is determined from the target detection frame.

[0021] In the above method, the adaptive interaction area that adapts to the target user's body shape is determined by determining the target detection frame. Compared with the method of determining the adaptive interaction area by identifying the user's body key points, since there is no need to identify and locate the relevant body key points, but to directly determine the target detection frame, the computational efficiency is improved.

[0022] In one embodiment, determining, from the target detection frame, an adaptive interaction area that includes the human interaction point and is adapted to the body shape of the target user includes: Determining a plurality of second candidate interaction areas that are adapted to the body shape of the target user from the target detection frame; Acquire second position information of the human body interaction point in the target user image; An adaptive interaction area to which the human body interaction point belongs is determined from a plurality of second candidate interaction areas according to the second position information.

[0023] In the above method, an adaptive interaction area is determined from multiple second candidate interaction areas according to the position of the human body interaction point, so that the determined adaptive interaction area matches the user's current interaction posture. In addition, since each second candidate interaction area is essentially a sub-area of the target detection frame area, the determined adaptive interaction area is also a sub-area of the target detection frame area. Compared with directly using the target detection frame area as the adaptive interaction area, the area of the adaptive interaction area is reduced. Since the relative position relationship between the human body interaction point and the adaptive interaction area corresponds to the relative position relationship between the mapping point and the front-end interaction page, the above method reduces the area of the adaptive interaction area so that the user can trigger corresponding controls on the front-end interaction page by making small movements within the adaptive interaction area, which is more convenient for user interaction operations and improves the user experience.

[0024] In one embodiment, obtaining the target user image includes: When the interaction request is an off-vehicle interaction request, the target user image is obtained by capturing an image of an area outside the vehicle through an off-vehicle image capture device of the vehicle; the front-end interaction page is a page projected and displayed by a vehicle projection device; and / or, When the interaction request is an in-vehicle interaction request, the target user image is obtained by capturing the image of the in-vehicle area through the in-vehicle image capture device of the vehicle; the front-end interaction page is a page displayed by the in-vehicle display device.

[0025] Through the above method, the application scenarios of human-computer interaction control are enriched. It is not only applicable to in-vehicle interaction scenarios, but also to out-of-vehicle interaction scenarios, thereby improving the usability and user experience of multi-scenario applications.

[0026] In one embodiment, before determining, from the target user image, an adaptive interaction area that includes the human body interaction point and is adapted to the target user's body shape, the method includes: Obtaining identity authentication information of the target user; Determine whether the identity verification information matches the preset legal identity information.

[0027] By using the above method, user identity authentication is performed before determining the adaptive interaction area of the target user, thereby ensuring the security of the interactive control behavior.

[0028] In one embodiment, determining a mapping point of the human body interaction point in the front-end interactive page based on the relative positional relationship between the human body interaction point of the target user and the adaptive interactive area includes: Acquire a relative positional relationship between a reference point in the adaptive interaction area and the human body interaction point in the target user image; The mapping point of the human body interaction point in the front-end interaction page is determined according to the reference mapping point of the reference point in the front-end interaction page and the relative position relationship.

[0029] In the above method, the mapping point of the human body interaction point in the front-end interaction page is determined by the relative position relationship between the reference point in the adaptive interaction area and the human body interaction point, which can achieve accurate mapping of the human body interaction point in the front-end interaction page and improve the accuracy of interactive control.

[0030] In one embodiment, determining the mapping point of the human body interaction point in the front-end interactive page according to the reference mapping point of the reference point in the front-end interactive page and the relative position relationship includes: Acquiring area size information of the adaptive interaction area; The mapping point of the human body interaction point in the front-end interaction page is determined according to the reference mapping point of the reference point in the front-end interaction page, the relative position relationship and the area size information.

[0031] In the above method, the mapping point of the human body interaction point in the front-end interactive page is determined by the reference mapping point of the reference point in the front-end interactive page, the relative position relationship between the reference point and the human body interaction point in the adaptive interactive area, and the area size information of the adaptive interactive area, thereby further improving the accuracy of the mapping point position determination.

[0032] In one embodiment, the area size information includes the width of the adaptive interaction area and height , determining the mapping point of the human body interaction point in the front-end interactive page according to the reference mapping point of the reference point in the front-end interactive page, the relative position relationship, and the area size information, including: according to Calculate the horizontal coordinate of the mapping point of the human body interaction point in the front-end interaction page ; according to Calculate the vertical coordinate of the mapping point of the human body interaction point in the front-end interaction page ; in, represents the position coordinates of the reference mapping point in the front-end interactive page, represents the displacement of the human body interaction point relative to the reference point in the horizontal axis direction of the target user image, represents the displacement of the human body interaction point relative to the reference point in the vertical coordinate axis direction of the target user image, and They respectively represent the first configuration parameter and the second configuration parameter configured for the front-end interactive page.

[0033] In the above method, the relative position between the human body interaction point and the reference point is adjusted in a corresponding proportion to adapt to the configuration of the front-end interaction page, further improving the accuracy of determining the mapping point position.

[0034] In one embodiment, when multiple frames of the target user image are acquired, interactive control of the front-end interactive page is implemented based on the mapping points, including: determining an operation trajectory of the target user based on mapping points corresponding to each of the target user images; Interactive control is achieved based on the operation trajectory.

[0035] In the above method, the corresponding operation trajectory is determined by the mapping point corresponding to each target user image, and interactive control is achieved through the corresponding operation trajectory, which expands the application scenario of achieving interactive control through air-to-air operation trajectories.

[0036] In one embodiment, when the front-end interactive page is a game page, implementing interactive control based on the operation trajectory includes: Determining whether the operation track falls into a corresponding determination area of a game element in the game page; The corresponding game feedback mechanism and / or game scoring mechanism is triggered according to the judgment result.

[0037] In the above method, by judging whether the operation track falls into the corresponding judgment area of the game element in the game page, the corresponding game mechanism is triggered according to the judgment result, thereby expanding the application scenario of realizing game interaction through the air operation track.

[0038] In a second aspect, the present application provides a human-computer interaction control device, the device comprising: An acquisition module, used to acquire the target user's image; a first determining module, configured to determine a human body interaction point of a target user from the target user image, and determine an adaptive interaction area from the target user image that includes the human body interaction point and is adapted to the body shape of the target user; A second determining module is configured to determine a mapping point of the human body interaction point in the front-end interactive page according to a relative positional relationship between the human body interaction point and the adaptive interactive area; A control module is used to implement interactive control of the front-end interactive page based on the mapping point.

[0039] In a third aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the human-computer interaction control method of the first aspect.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human-computer interaction control method of the first aspect.

[0041] The human-computer interaction control method, device, electronic device and storage medium provided by the present application obtain a target user image upon receiving an interaction request, determine the target user's human body interaction points from the target user image, and determine an adaptive interaction area from the target user image that includes the human body interaction points and is adapted to the target user's body shape. According to the relative positional relationship between the human body interaction points and the adaptive interaction area, the mapping points of the human body interaction points in the front-end interaction page are determined, and air-to-air interaction control of the front-end interaction page is realized based on the mapping points. In addition, by determining the adaptive interaction area adapted to the target user's body shape, the problem of insufficient response stability due to large differences in body shape among different users and the mismatch with the user's interaction intention can be avoided, thereby improving the interaction accuracy and user experience.

[0042] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or may be learned through practice of the present application. The objectives and other advantages of the present application may be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. It should be understood that the above general description and the detailed description that follow are merely exemplary and explanatory and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 1 is a flow chart of human-computer interaction control in one embodiment; Figure 2 is a schematic diagram of the distribution of first candidate interaction areas in one embodiment; Figure 3 is a schematic diagram of a first positional relationship between a human body interaction point and a first candidate interaction area in one embodiment; Figure 4 is a schematic diagram of a second positional relationship between a human body interaction point and a first candidate interaction area in one embodiment; Figure 5 is a schematic diagram of a second positional relationship between a human body interaction point and a first candidate interaction area in one embodiment; Figure 6 is a schematic diagram of a first interaction gesture of a target user in one embodiment; Figure 7 is a schematic diagram of a second interaction posture of a target user in one embodiment; Figure 8 A schematic diagram of a process for determining a mapping point in one embodiment; Figure 9 Schematic diagram of the structure of a human-computer interaction control device in one embodiment; Figure 10 FIG. 1 is a schematic structural diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] This application embodiment provides a human-computer interaction control method, see Figure 1 As shown, including: S11: When receiving an interaction request, obtain a target user image.

[0047] S12: determining a human body interaction point of the target user from the target user image, and determining an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user from the target user image.

[0048] S13: Determine a mapping point of the human body interaction point in the front-end interaction page according to a relative position relationship between the human body interaction point and the adaptive interaction area.

[0049] S14: Implement interactive control of the front-end interactive page based on the mapping point.

[0050] The human-computer interaction control method in the embodiments of the present application can be applied to electronic devices, which can be mobile devices such as mobile phones, tablets, or other wearable devices. The electronic devices can also be other devices that can interact with users and display relevant pages, such as AR (Augmented Reality) devices or in-vehicle devices. In addition, the human-computer interaction control method in the embodiments of the present application can be applied to interactive games such as "fruit-cutting" motion-sensing games. For example, by detecting the movement trajectory of the player's hand, direct interaction with virtual objects can be achieved, which can enhance the user's immersive experience.

[0051] The human-computer interaction control method provided in the embodiment of the present application determines the mapping point of the human body interaction point in the front-end interaction page according to the relative position relationship between the human body interaction point in the target user image and the adaptive interaction area, and realizes the remote interaction control of the front-end interaction page based on the mapping point. Since the adaptive interaction area is an area adapted to the body shape of the target user, it can avoid the problem of insufficient response stability or mismatch with the user's interaction intention due to large differences in body shape between different users, thereby improving the interaction accuracy. In addition, compared with directly determining the mapping point of the human body interaction point based on the relative position relationship of the target user's human body interaction point with respect to the entire collected image, since the body shape of the target user is taken into consideration, the target user only needs a small movement to achieve interactive control, thereby improving the user experience.

[0052] In one embodiment, the human-computer interaction control method is applied to a human-vehicle interaction scenario. When the interaction request is an off-vehicle interaction request, the vehicle's off-vehicle image acquisition device performs image acquisition of the area outside the vehicle to obtain a target user image, and the front-end interaction page is a page projected and displayed by the vehicle's projection device; and / or, when the interaction request is an in-vehicle interaction request, the vehicle's in-vehicle image acquisition device performs image acquisition of the in-vehicle area to obtain a target user image, and the front-end interaction page is a page displayed by the in-vehicle display device.

[0053] In an embodiment of the present application, the vehicle can receive an interaction request sent by a user, and the manner in which the interaction request is sent includes but is not limited to at least one of voice, gesture, and button input.

[0054] The vehicle projection device in the embodiment of the present application refers to a device that can project on the target projection area of the vehicle. The target projection area can be the area outside the vehicle or the area of the vehicle body. The developer can pre-set the position of the target projection area. Of course, the position can also be customized by the user.

[0055] Exemplarily, the vehicle projection device is a vehicle headlight projection device. In this case, the target projection area is the area outside the vehicle. The vehicle headlight projection device projects relevant information in the area outside the vehicle so that the target user outside the vehicle can implement interactive control based on the projected information. For example, the front-end interactive page can be projected onto a wall or a projection screen through the vehicle headlight projection device.

[0056] Exemplarily, the vehicle projection device is a front windshield display device. In this case, the target projection area is the front windshield area, that is, the vehicle body area, and the relevant information is displayed externally through the front windshield display device.

[0057] The in-vehicle display device in the embodiment of the present application can be a vehicle-mounted display screen, or of course, a front windshield display device. In this case, relevant information can be displayed internally through the front windshield.

[0058] To ensure the security and legality of the interactive control behavior, in one embodiment, before step S12, the following steps may be further included: Obtain the target user's authentication information; Verify that the authentication information matches the preset legal identity information.

[0059] The identity authentication information in the embodiment of the present application includes but is not limited to at least one of facial information, voice information and input password information. When it is determined that the facial information of the target user matches the preset facial information, and / or when it is determined that the voice information of the target user matches the preset voice information, and / or when it is determined that the password information input by the target user matches the preset password information, step S12 is executed.

[0060] It should be noted that the target user image in the embodiment of the present application refers to an image containing the target user. When there are multiple character objects in the target user image, that is, when the target user image contains other character objects in addition to the target user, the target user can be determined from the target user image, so that the human body interaction points and adaptive interaction areas are only determined for the target user in the future, and there is no need to establish human body interaction points and adaptive interaction areas for other character objects, thereby saving computing resources and improving the efficiency of human-computer interaction control.

[0061] Next, a method of determining a target user from a target user image is introduced.

[0062] In one embodiment, facial information of each person in the target user's image can be obtained, and the person whose facial information matches the preset facial information is selected as the target user. In this embodiment, by obtaining facial information, not only can the target user be determined, but the legitimacy of the target user can also be guaranteed.

[0063] In one embodiment, for each person object in the target user's image, a detection frame is determined that covers each person object. Based on the distribution of each detection frame, a target detection frame is determined from the multiple detection frames, and the person object in the target detection frame is used as the target user. For example, the detection frame closest to the center of the target user's image can be used as the target detection frame; alternatively, the detection frame with the largest area can be used as the target detection frame.

[0064] In this embodiment, the detection frame can be determined by the target detection model. It should be noted that the target detection model in the embodiment of the present application is used to generate a detection frame that can cover the corresponding character object for each character object in the target user image. The structure of the target detection model in the embodiment of the present application may include but is not limited to at least one of FCOS (Fully Convolutional One-Stage Object Detection), R-CNN (Region-based Convolutional Neural Networks), YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and deformable convolutional network.

[0065] For a continuous interaction scenario, multiple frames of target user images can be obtained. At this time, after the target user is determined for the first time, the target user can be locked. In this way, there is no need to determine the target user for each subsequent frame of target user image using the above method, which can improve operating efficiency.

[0066] Taking the example of determining the target user through facial information, when the first frame of the target user image is obtained, the target user is determined from the target user image based on the facial information, and the target user is locked. The locking method can be as follows: Obtain the position area of the target user in the target user image, and for each subsequent frame of the target user image received, directly use the human object in the position area as the target user.

[0067] The above method of targeting the target user by location area is suitable for static location interaction scenarios. Considering that the target user's location may change during the interaction process, to further improve the accuracy of target user identification during the entire interaction process, after determining the target user through facial information, the target user can also be targeted by the following methods: The target user's clothing information is obtained. For each subsequently received frame of the target user's image, the target user is identified as the target user if the clothing of the person in the target user's image matches the clothing indicated by the clothing information. The target user's clothing information includes, but is not limited to, clothing color information and / or clothing element information of the clothing worn by the target user. The clothing element information indicates the pattern type of the clothing.

[0068] Since face recognition has higher requirements for details than clothing recognition, it usually requires more computing resources. In the above method, the target user is first determined by facial information, and then the target user is locked by clothing information. Compared with extracting and recognizing facial information for each frame of the target user image, more computing resources can be released and operating efficiency can be improved. In addition, locking the target user by clothing information allows the target user to interact with the image acquisition device with his back to the subsequent interaction process, which expands more application scenarios and improves the user experience.

[0069] In one embodiment, for a continuous interaction scenario, the target user may also be determined in the following manner: Action recognition can be performed on the multiple frames of target user images collected, and the person object that first performs the preset target action is used as the target user, and the target user is locked. For each subsequent frame of the target user image received, the human body interaction point and adaptive interaction area of the target user are determined to achieve interactive control. The method of locking the target user in this embodiment can refer to the locking method introduced above and will not be repeated here. The preset target action can be flexibly set by the developer and also supports user customization, such as raising a hand or making a heart gesture.

[0070] Next, the above step S12 is described in detail.

[0071] First, a method for determining a target user's human body interaction point from a target user image is introduced. The human body interaction point in the embodiment of the present application represents a point of interaction with an electronic device, and the human body interaction point can be one or more.

[0072] In one embodiment, the target user can interact with the aforementioned electronic devices through an interactive device worn by the target user. The location of the interactive device can be identified and used as a human interaction point. The target user can wear the interactive device anywhere on the body, such as the wrist or elbow. It will be appreciated that in this embodiment, the target user in the target user image can also be identified and locked by identifying the interactive device. For example, when there are multiple human subjects in the target user image, the human subject wearing the interactive device can be identified as the target user.

[0073] In one embodiment, first position information of key points of the target user's body in the target user image may be determined, and the human body interaction points may be determined from the target user image based on the first position information.

[0074] Exemplarily, the human body key points of the target user in the target user image can be identified by a human body key point recognition model to obtain first position information of the human body key points. The human body key points identified by the human body key point recognition model can be directly used as human body interaction points. In this case, the position of the human body key point is also the position of the human body interaction point. For example, the identified left wrist key point and / or right wrist key point can be used as human body interaction points. In this case, the position of the left wrist key point and / or right wrist key point is also the position of the human body interaction point. Of course, the first position information of the human body key point can also be further processed to obtain the human body interaction point. For example, the first position information of the wrist key point and the first position information of the finger key point can be identified by the human body key point recognition model, and then the position of the palm key point is determined based on the first position information of the wrist key point and the first position information of the finger key point, and the position is used as the position of the human body interaction point.

[0075] Next, a method for determining an adaptive interaction area adapted to the target user's body shape from the target user's image is described. It should be noted that the adaptive interaction area in the embodiment of the present application is the area containing the human body interaction point, that is, the human body interaction point is located within the adaptive interaction area, and the adaptive interaction area in the embodiment of the present application is a partial image area of the entire image area of the target user's image, which is adapted to the target user's body shape.

[0076] In one embodiment, an adaptive interaction area can be generated based on the location of the human interaction point and pre-set adaptive interaction area generation rules. For example, the location of the human interaction point can be used as the center point of the rectangle. If there are two human interaction points, the center point of the two human interaction points can be used as the center point of the rectangle. Based on the center point of the rectangle, the target length, and the target width, a target rectangular area is determined from the target user image and used as the adaptive interaction area.

[0077] For example, the target length and target width can be calculated as follows: Obtaining the distance between the target user and the image acquisition device; Determine the target length and target width based on the spacing.

[0078] It can be understood that since the spacing will affect the image area occupied by the target user in the target user image, in this embodiment, the target length and target width of the adaptive interaction area are determined based on the spacing between the target user and the image acquisition device, and then the adaptive interaction area is determined according to the target length and target width, so that the determined adaptive area is adapted to the image area occupied by the target user in the target user image, that is, the adaptive interaction area is adapted to the body shape of the target user, which can avoid incorporating areas unrelated to the target user into the adaptive interaction area, making the determined adaptive interaction area more reasonable.

[0079] For example, the formula Calculate the target length according to the formula Calculate the target width, where and represent the target length and target width respectively, Indicates the distance between the target user and the image acquisition device, and Represents the physical length and physical width respectively, and Represent the third configuration parameter and the fourth configuration parameter respectively.

[0080] The physical length and physical width refer to the length and width in physical space. These values can be pre-set fixed values. For example, developers can set the physical length and physical width based on users with a well-proportioned build, such as using the length and width of the matrix area in physical space that a well-proportioned user's hand can reach as the physical length and physical width. Of course, these values can also be determined based on the current target user's body characteristics, such as using the current target user's height as the physical width and the current target user's width (such as the distance between the two shoulders) as the physical length.

[0081] The third configuration parameter and the fourth configuration parameter can be fixed values pre-set based on the camera parameters of the image acquisition device. For example, the value can be set according to the camera angle and focal length of the image acquisition device; of course, the third configuration parameter and the fourth configuration parameter can also be values determined based on the body characteristics of the target user.

[0082] It is understandable that when and When it is a preset fixed value, in order to make the obtained adaptive interaction area more adapted to the body shape of the target user, a first body feature parameter that can reflect the target user's fatness or thinness and a second body feature parameter that can reflect the target user's height can be obtained. According to a first corresponding relationship between a preset first body feature parameter range and a preset third configuration parameter, the target first body feature parameter range to which the target user's first body feature parameter belongs is determined, and the third configuration parameter corresponding to the target first body feature parameter range is used as the third configuration parameter corresponding to the target user's first body feature parameter, and is substituted into the formula for calculating the target length to calculate the target length.

[0083] Similarly, the target second body shape feature parameter range to which the second body shape feature parameter of the target user belongs can be determined based on the second corresponding relationship between the preset second body shape feature parameter range and the preset fourth configuration parameter, and the fourth configuration parameter corresponding to the target second body shape feature parameter range is used as the fourth configuration parameter corresponding to the second body shape feature parameter of the target user, and is substituted into the formula for calculating the target width to calculate the target width.

[0084] In the embodiment of the present application, the first body shape characteristic parameter may be the maximum width of the target user, and the second body shape characteristic parameter may be the height of the target user.

[0085] It can be understood that in the first correspondence, the larger the preset first body feature parameter is, the larger the corresponding third configuration parameter is; in the second correspondence, the larger the preset second body feature parameter is, the larger the corresponding fourth configuration parameter is. In this way, for target users with larger bodies, a larger adaptive interaction area can be adaptively determined; for target users with smaller bodies, a smaller adaptive interaction area can be adaptively determined, thereby achieving the effect of automatically adapting the appropriate adaptive interaction range according to the body characteristics of different users, and improving the usability and user experience in multiple users and multiple scenarios.

[0086] In one embodiment, similar to the above-mentioned method of determining the human body interaction points, the first position information of the human body key points of the target user in the target user image can also be determined, and an adaptive interaction area that includes the human body interaction points and is adapted to the target user's body shape can be determined from the target user image based on the first position information.

[0087] It should be noted that if the human body interaction points are obtained through human body key point identification, the identification results at this time may include other human body key points in addition to the above-mentioned human body key points used to determine the human body interaction points, so as to determine the adaptive interaction area through the first position information of other human body key points.

[0088] In one embodiment, multiple first candidate interaction regions adapted to the target user's body type can be determined from the target user's image based on the first position information of the human body key point. Second position information of the human body interaction point in the target user's image is obtained, and the adaptive interaction region to which the human body interaction point belongs is determined from the multiple first candidate interaction regions based on the second position information. It will be understood that if a recognized human body key point is directly used as the human body interaction point, the second position information is also the first position information.

[0089] The human key point recognition model in the embodiment of the present application can be a model obtained through training, and the structure of the model may include but is not limited to at least one of CNN (Convolutional Neural Network), OpenPose (pose estimation), YOLO (You Only Look Once, single-stage target detector) and HRNet (High-Resolution Net).

[0090] Exemplarily, the identified key points of the human body may include a first key point, and the electronic device determines multiple first candidate interaction areas based on the first key point. The determined multiple first candidate interaction areas include a top interaction area located between the first key point and the upper interaction boundary line, and a lower interaction area located between the first key point and the lower interaction boundary line.

[0091] For example, see Figure 2 As shown, the two longitudinal boundary lines corresponding to each first candidate interaction area can be determined, and the area enclosed by the first horizontal line where the first key point is located, the two longitudinal boundary lines and the upper interaction boundary line is determined as the top interaction area, and the area enclosed by the first horizontal line and the other two longitudinal boundary lines and the lower interaction boundary line is determined as the lower interaction area. At this time, the first horizontal line is the dividing line between the top interaction area and the lower interaction area. In order to avoid confusion in the mapping logic when the human body interaction point is subsequently mapped, the pixel area corresponding to the dividing line can be used as part of the top interaction area or the lower interaction area. For example, if the dividing line is included in the top interaction area, then when it is detected that the human body interaction point is located on the dividing line, it is determined that the human body interaction point is located in the top interaction area, and the top interaction area can be used as the current adaptive interaction area. Assuming that the human body interaction point is the wrist key point, with Figure 2 For example, since the wrist key point is located in the lower interaction area, the lower interaction area is used as the adaptive interaction area.

[0092] Among them, the upper interaction boundary line represents the boundary corresponding to the highest limit position that the target user's human interaction point can reach upward during the interaction process (such as the limit position that the wrist key point can reach when the hand is raised upward), and the lower interaction boundary line represents the boundary corresponding to the lowest limit position that the target user's human interaction point can reach downward during the interaction process (such as the limit position that the wrist key point can reach when the hand is naturally hanging down). The two longitudinal boundary lines represent the boundaries corresponding to the farthest positions that the target user's human interaction point can reach in the horizontal directions to the left and right during the interaction process. For example, it can be the limit position that the left wrist key point can reach to the left when the hand is horizontal with the shoulder, and the limit position that the right wrist key point can reach to the right.

[0093] Since the positions of the first key points corresponding to users of different body shapes are different, the position of the first candidate interaction area obtained by the above method is related to the user's body shape, which can meet the requirements of users of different body shapes for the interaction area and improve the adaptability of the first candidate interaction area to the user's body shape.

[0094] In addition, since each first candidate interaction area is actually a sub-area of the entire area between the upper interaction boundary and the lower interaction boundary line, the determined adaptive interaction area is also a sub-area of the entire area between the upper interaction boundary and the lower interaction boundary line. Compared with directly using the entire area between the upper interaction boundary and the lower interaction boundary line as the adaptive interaction area, the area of the adaptive interaction area is reduced. Since the relative position relationship between the human body interaction point and the adaptive interaction area corresponds to the relative position relationship between the mapping point and the front-end interaction page, the above method reduces the area of the adaptive interaction area so that the user can trigger corresponding controls on the front-end interaction page by making small movements within the adaptive interaction area, which is more convenient for user interaction operations and improves the user experience.

[0095] For example, see Figure 3 As shown, the identified human body key points include not only the above-mentioned first key points, but also a second key point located below the first key point. The lower interaction area may include an intermediate interaction area between the first key point and the second key point and a bottom interaction area between the second key point and the lower interaction boundary line. Therefore, this example includes at least three first candidate interaction areas: the top interaction area, the middle interaction area, and the bottom interaction area. Compared with the two first candidate interaction areas determined in the above example, the interaction area is divided more finely, which further enhances the user's interaction experience. At this time, the adaptive interaction area determined is one of the top interaction area, the middle interaction area, and the bottom interaction area.

[0096] As an example, the first key point is the top of the head key point, and the second key point is the shoulder key point. Regardless of whether the target user is facing the image acquisition device or facing away from the image acquisition device, the top of the head key point and the shoulder key point can be accurately located, solving the problem of human body key point occlusion when the target user faces away from the image acquisition device for air interaction. Even if the target user faces away from the image acquisition device, the appropriate adaptive interaction area can be accurately determined, thereby improving the user's interactive experience. In actual applications, users generally tend to interact in the middle interaction area. This embodiment effectively improves the robustness and response accuracy of the middle interaction area by extending the candidate interaction area upward to above the top of the head.

[0097] Taking the palm key point as an example, when the palm key point is located in the bottom interaction area, as shown in Figure 3 As shown, the bottom interactive area is determined as the corresponding adaptive interactive area. Figure 4 As shown in , when the palm key point is located in the top interaction area, the top interaction area is determined as the corresponding adaptive interaction area. Figure 5 As shown, when the palm key point is located in the middle interaction area, the middle interaction area is determined as the corresponding adaptive interaction area.

[0098] It should be noted that, when there are multiple human body interaction points, for example, a first human body interaction point and a second human body interaction point, interactive control can be achieved in the following manner.

[0099] As a first example, for each human interaction point, an adaptive interaction area that can include the human interaction point and adapt to the target user's body shape can be determined, and the relative position relationship between each human interaction point and its corresponding adaptive interaction area can be determined, and based on each relative position relationship, the corresponding mapping point of each human interaction point can be determined. For example, when the target user presents Figure 6 In the interactive posture shown, since the first human body interaction point is located in the middle interaction area and the second human body interaction point is located in the bottom interaction area, the middle interaction area can be used as the adaptive interaction area corresponding to the first human body interaction point, and the bottom interaction area can be used as the adaptive interaction area corresponding to the second human body interaction point, and the corresponding mapping points are determined according to the relative position relationship between each human body interaction point and its corresponding adaptive interaction area.

[0100] As a second example, the adaptive interaction area can be determined based on the union area of the first candidate interaction areas to which each human interaction point belongs. Specifically, if the first candidate interaction areas to which each human interaction point belongs are connected, the union area of the first candidate interaction areas to which each human interaction point belongs is directly used as the adaptive interaction area. If the first candidate interaction areas to which each human interaction point belongs are not connected, the connected area connecting the first candidate interaction areas to which each human interaction point belongs is obtained, and the union area of the first candidate interaction areas to which each human interaction point belongs and the connected area is used as the adaptive interaction area.

[0101] In this example, the relative position relationship of each human interaction point relative to the same union area (i.e., the adaptive interaction area) is determined, and the mapping points corresponding to each human interaction point are determined based on the relative position relationship. Figure 6 Take the following example to illustrate: Figure 6 In the interactive gesture shown, since the first human interaction point is located in the middle interactive area and the second human interaction point is located in the bottom interactive area, the middle interactive area and the bottom interactive area are connected, the union area of the middle interactive area and the bottom interactive area can be directly used as the adaptive interactive area corresponding to the first human interaction point and the second human interaction point. Figure 7 In the interactive posture shown, since the first human body interaction point is located in the top interaction area and the second human body interaction point is located in the bottom interaction area, the top interaction area and the bottom interaction area are not connected. At this time, a connecting area can be obtained to connect the top interaction area and the bottom interaction area. For example, at this time, the overall area composed of the top interaction area, the middle interaction area and the bottom interaction area can be used as the adaptive interaction area of the first human body interaction point and the second human body interaction point.

[0102] As a third example, an adaptive interaction area can be directly determined that can simultaneously include various human body interaction points and adapt to the body shape of the target user.

[0103] The method for determining the interactive upper boundary line in the embodiment of the present application includes but is not limited to at least one of the following methods: Method 1: Determine a target detection frame covering the target user from the target user image; and determine the upper detection frame line of the target detection frame as the upper interaction boundary line.

[0104] Method 2: The key points of the human body include the third key point, a first height characteristic parameter of the target user is obtained, and the upper boundary line of interaction is determined according to the first height characteristic parameter and the first position information of the third key point.

[0105] Method three: the human body key points include a third key point, and the upper interaction boundary line is determined according to the first position information of the third key point and the first preset distance parameter.

[0106] In the above-mentioned method 1, the target detection frame of the target user can be determined according to the target detection model introduced above, which will not be described in detail here.

[0107] In the above-mentioned second method, the first height characteristic parameter refers to a parameter that can reflect the height characteristic between the first pre-designated body parts of the target user.

[0108] Exemplarily, the human body key points in the target user image include the fifth key point and the sixth key point. The first height feature parameter can be determined based on the first position information of the fifth key point and the first position information of the sixth key point. For example, the distance between the fifth key point and the sixth key point can be used as the first height feature parameter. At this time, the first height feature parameter reflects the height feature between the fifth key point and the sixth key point. Then, the upper boundary line of interaction is determined based on the product of the first height feature parameter and the first preset height ratio coefficient and the first position information of the third key point.

[0109] For example, the fifth key point is the shoulder key point, and the sixth key point is the waist key point. Then, the sixth key point can be placed above the third key point and at a distance of The horizontal line of is used as the upper boundary line of interaction, where represents the first preset height scale factor, Indicates the distance between the shoulder key point and the waist key point in the longitudinal direction (vertical direction), which is also the first height feature parameter. As an example, the third key point can be the waist key point, The value of can be 1.6. In this case, the distance from the waist key point above and in the longitudinal direction to the waist key point can be The horizontal line is used as the upper boundary line of the interaction.

[0110] In the above method three, the first preset height coefficient can be flexibly set by the developer. In this method, a horizontal line above the third key point and at a distance from the third key point in the vertical direction by the first preset distance parameter can be used as the upper boundary line of the interaction.

[0111] The method for determining the interaction lower boundary line in the embodiment of the present application includes but is not limited to at least one of the following methods: Method 4: The human body key points include the fourth key point, the second height characteristic parameter of the target user is obtained, and the interaction lower boundary line is determined according to the second height characteristic parameter and the first position information of the fourth key point.

[0112] Method 5: The human body key points include a fourth key point, and the interaction lower boundary line is determined according to the first position information of the fourth key point and the second preset distance parameter.

[0113] In the fourth method above, the second height characteristic parameter refers to a parameter that can reflect the height characteristic between the second pre-designated body parts of the target user.

[0114] Exemplarily, the second height feature parameter is different from the first height feature parameter, and the human body key points in the target user image include the seventh key point and the eighth key point. The second height feature parameter can be determined based on the first position information of the seventh key point and the first position information of the eighth key point. For example, the distance between the seventh key point and the eighth key point can be used as the second height feature parameter. At this time, the second height feature parameter reflects the height feature between the seventh key point and the eighth key point. Then, the interaction lower boundary line is determined based on the product of the second height feature parameter and the second preset height ratio coefficient and the first position information of the fourth key point.

[0115] For example, the second height characteristic parameter is the same as the first height characteristic parameter, that is, the fifth key point is the same as the seventh key point, and the sixth key point is the same as the eighth key point. The fifth key point is the shoulder key point, and the sixth key point is the waist key point. The horizontal line of is used as the lower boundary line of interaction, where Indicates the second preset height scale factor. The value of can be 0.3. In this case, the distance from the waist key point below and in the longitudinal direction to the waist key point can be The horizontal line is used as the lower boundary of the interaction.

[0116] In the above-mentioned method five, the second preset height coefficient can be flexibly set by the developer. In this method, a horizontal line above the fourth key point and at a distance from the fourth key point in the vertical direction by the second preset distance parameter can be used as the lower boundary line of the interaction.

[0117] In the embodiment of the present application, each candidate interaction area may be located between two longitudinal boundary lines. The longitudinal boundary lines may be determined in a manner including but not limited to at least one of the following manners: Method 6: Determine a target detection frame covering the target user from the target user image, and determine the vertical detection frame line of the target detection frame as the vertical boundary line.

[0118] Method 7: The human body key points in the target user image include the ninth key point and the tenth key point, and the horizontal interaction limit parameter of the target user is obtained. The first longitudinal boundary line is determined based on the ninth key point and the horizontal interaction limit parameter, and the second longitudinal boundary line is determined based on the tenth key point and the horizontal interaction limit parameter.

[0119] In the sixth method described above, the target detection frame of the target user can be determined based on the target detection model introduced above, which will not be described in detail here.

[0120] The horizontal interaction limit parameter in the seventh method above reflects the maximum distance that the target user can reach in the horizontal direction during the interaction process. The horizontal interaction limit parameter can be determined based on key points of the human body or can be pre-set by the developer.

[0121] For example, the distance between the left shoulder / right shoulder key point and the left finger / right finger key point when the hand is in a straight and stretched state can be obtained as the horizontal interaction limit parameter. The ninth key point and the tenth key point are the left shoulder key point and the right shoulder key point respectively. Then, the vertical line at the position where the left shoulder key point is horizontally to the left and the distance from the left shoulder key point is the horizontal interaction limit parameter can be used as the first longitudinal boundary line, and the vertical line at the position where the right shoulder key point is horizontally to the right and the distance from the right shoulder key point is the horizontal interaction limit parameter can be used as the second longitudinal boundary line.

[0122] In one embodiment, the human body key points used to determine human interaction points are non-rigid key points, that is, key points corresponding to human body parts whose position is easily changed during interaction. For example, they may include but are not limited to at least one of the wrist key points, finger key points, or elbow key points. To ensure the stability of the determined first candidate interaction area, the human body key points used to determine the first candidate interaction area are rigid key points, that is, key points corresponding to human body parts whose position is not easily changed during interaction. For example, they may include but are not limited to at least one of the top of the head key points, shoulder key points, and waist key points.

[0123] In one embodiment, the adaptive interaction area can also be determined by determining a target detection frame. Specifically, a target detection frame encompassing the human interaction points and the target user is determined from the target user image. From the target detection frame, an adaptive interaction area that includes the human interaction points and is adapted to the target user's body shape is determined. In this embodiment, the target detection frame can be determined using the target detection model described above, and further details will not be given here.

[0124] In this embodiment, the adaptive interaction area adapted to the target user's body shape is determined by determining the target detection frame. Compared with determining the adaptive interaction area by identifying the user's body key points, since there is no need to identify and locate multiple relevant body key points, but the target detection frame is directly determined, the calculation efficiency is improved.

[0125] In one embodiment, the entire area within the target detection frame may be directly used as the adaptive interaction area.

[0126] In one embodiment, multiple second candidate interaction areas that are adapted to the body shape of the target user are determined from the target detection frame, second position information of the human body interaction point in the target user image is obtained, and the adaptive interaction area to which the human body interaction point belongs is determined from the multiple second candidate interaction areas based on the second position information.

[0127] Exemplarily, the target detection frame can be divided into two second candidate interaction areas distributed in an upper and lower manner, and the second candidate interaction area where the human body interaction point is currently located is used as an adaptive interaction area. Of course, the target detection frame can also be divided into three or more second candidate interaction areas. When dividing the target detection frame, it can be divided evenly, or it can be divided in other ways. For example, the area corresponding to the target detection frame is evenly divided into 4 sub-areas arranged vertically. Since the human body interaction point has a greater probability of moving in the area corresponding to the upper half of the body during the interactive control process, the topmost sub-area can be used as a second candidate interaction area, and the two middle sub-areas can be used as a second candidate interaction area. That is, by dividing the area corresponding to the target detection frame into 4 equal parts, 2 second candidate interaction areas are obtained.

[0128] In this embodiment, an adaptive interaction area is determined from multiple second candidate interaction areas based on the position of the human body interaction point, so that the determined adaptive interaction area matches the user's current interaction posture. In addition, since each second candidate interaction area is essentially a sub-area of the target detection frame area, the determined adaptive interaction area is also a sub-area of the target detection frame area. Compared with directly using the target detection frame area as the adaptive interaction area, the area of the adaptive interaction area is reduced. Since the relative position relationship between the human body interaction point and the adaptive interaction area corresponds to the relative position relationship between the mapping point and the front-end interaction page, the above method reduces the area of the adaptive interaction area, so that the user can trigger corresponding controls on the front-end interaction page by making small movements within the adaptive interaction area, which is more convenient for user interaction operations and improves the user experience.

[0129] It should be noted that the above contents are all illustrative examples using the first candidate interaction area and the second candidate interaction area as rectangles, and do not constitute a limitation on the embodiments of the present application. In some embodiments, the first candidate interaction area and the second candidate interaction area may also be other shapes.

[0130] Next, step S13 is described in detail.

[0131] See Figure 8 As shown, step S13 may include the following sub-steps: S131: Obtain the relative position relationship between the reference point in the adaptive interaction area and the human body interaction point in the target user image.

[0132] S132: Determine a mapping point of the human body interaction point in the front-end interactive page according to a reference mapping point and a relative position relationship of the reference point in the front-end interactive page.

[0133] Developers can pre-set reference point determination rules and pre-set corresponding reference mapping points for the front-end interactive page based on the reference point determination rules.

[0134] In step S131, a reference point can be determined from the adaptive interaction area according to a pre-set reference point determination rule. For example, the left vertex of the adaptive interaction area can be used as the reference point, and the reference mapping point pre-set for the front-end interaction page can be the left vertex of the front-end interaction page. For example, the center point of the adaptive interaction area can be used as the reference point, and the reference mapping point pre-set for the front-end interaction page can be the center point of the front-end interaction page.

[0135] In an embodiment of the present application, the relative position relationship between the reference point and the human body interaction point reflects the relative position relationship between the reference mapping point and the mapping point of the human body interaction point in the front-end interaction page. Therefore, based on the relative position relationship between the reference point and the human body interaction point and the reference mapping point, the mapping point of the human body interaction point can be determined from the front-end interaction page.

[0136] It should be noted that the front-end interactive page in the embodiment of the present application is a page area in the front-end page for user operations, which can be a partial page area in the entire front-end page or the entire front-end page area.

[0137] In order to ensure the accuracy of the position of the determined mapping point and thereby improve the accuracy of interactive control, in one embodiment, for the above-mentioned step S132, the area size information of the adaptive interaction area can be obtained, and the mapping point of the human body interaction point in the front-end interactive page can be determined based on the reference mapping point, relative position relationship and area size information of the reference point in the front-end interactive page.

[0138] The area size information in the embodiment of the present application may include the width of the adaptive interaction area and height , determining the mapping point of the human body interaction point in the front-end interactive page based on the reference mapping point, relative position relationship, and area size information of the reference point in the front-end interactive page may include: according to Calculate the horizontal coordinate of the mapping point of the human interaction point in the front-end interactive page ;according to Calculate the vertical coordinate of the mapping point of the human interaction point in the front-end interactive page .

[0139] in, Indicates the position coordinates of the reference mapping point in the front-end interactive page. Indicates the displacement of the human interaction point relative to the reference point in the horizontal axis direction of the target user image, Indicates the displacement of the human interaction point relative to the reference point in the vertical axis direction of the target user image. and They respectively represent the first configuration parameter and the second configuration parameter configured for the front-end interactive page.

[0140] If the position coordinates of the reference point in the target user image are , the position coordinates of the human interaction point in the target user image are , then the displacement of the human interaction point relative to the reference point in the horizontal axis direction of the target user image is Expressed as , The positive or negative value of represents the direction of the human body interaction point relative to the reference point on the horizontal axis. represents the distance between the human interaction point and the reference point in the horizontal axis direction. Then the displacement of the human interaction point relative to the reference point in the vertical axis direction of the target user image is Expressed as , The positive or negative value of represents the direction of the human body interaction point relative to the reference point on the vertical axis. Indicates the distance between the human body interaction point and the reference point in the direction of the vertical axis.

[0141] and It can be a preset screen resolution parameter. For example, if the resolution of the screen currently displaying the front-end interactive page is 1920×1080, then The value of is 1920. The value is 1080.

[0142] Regarding step S14, in one embodiment, the target cursor can be controlled to be displayed at the location of the mapping point. In this case, the position of the target cursor is controlled based on the mapping point. For example, it can be applied to teaching and other related scenarios, such as providing prompts to the students based on the location of the target cursor. In addition, the electronic device can trigger a pre-set corresponding control strategy based on the current location of the target cursor. For example, when the target cursor is located on a specific button on the front-end interactive page, a corresponding specific function is triggered.

[0143] Generally speaking, interactive control is a continuous process. Therefore, in one embodiment, when multiple frames of target user images are acquired, the operation trajectory of the target user can be determined based on the mapping points corresponding to each target user image, and interactive control can be implemented based on the operation trajectory.

[0144] It is understandable that, for each frame of the target user image, after obtaining the mapping point position information of the mapping point of the human body interaction point in each target user image, the corresponding operation trajectory can be determined according to the mapping point position information.

[0145] For example, N consecutive frames of target user images can be used as a unit. Each time N frames of target user images are obtained, the target user's operation trajectory is determined based on the mapping points corresponding to the N target user images. The value of N can be flexibly set by the developer. To provide prompts to the target user, the corresponding operation trajectory can also be displayed on the front-end interactive page. Specifically, after each N frames of target user images are obtained, the electronic device can package the location information of the mapping points corresponding to the N frames of target user images and send it to the front-end interactive page to display the corresponding operation trajectory on the front-end interactive page. Exemplarily, the corresponding operation track may be determined based on a preset continuous time length as a unit and according to a mapping point of the target user image corresponding to the preset continuous time length.

[0146] For example, the operation trajectory can be matched with a trajectory library, which stores the correspondence between preset operation trajectories and preset interaction control strategies. Based on the target user's operation trajectory and this correspondence, the target interaction control strategy corresponding to the operation trajectory is determined, and the corresponding interaction control is implemented according to the target interaction control strategy. Both the preset operation trajectory and the preset interaction control strategy can be flexibly set by the developer based on the actual application scenario, and of course, user customization is also supported. As an example, the interaction control strategy can be switching pages or returning to a page.

[0147] Exemplarily, when the front-end interactive page is a game page, implementing interactive control based on the operation trajectory includes: determining whether the operation trajectory falls into the corresponding judgment area of the game element in the game page; triggering the corresponding game feedback mechanism and / or game scoring mechanism based on the judgment result.

[0148] In the embodiments of the present application, the game elements have corresponding decision areas. The size and location of these decision areas can be flexibly configured by the developer based on actual application needs. For example, they can be an area on the game page that just covers the game element. The decision areas can also be set with corresponding validity periods. For example, the timer can be set from the display time of the game element, and the time period within the preset timer is the validity period of the decision area.

[0149] After obtaining the mapping point position information of each mapping point, the mapping point position information of the mapping point can be bound to the corresponding interaction operation time point. For example, the generation time of the mapping point position information can be used as the interaction operation time point. In this way, for each operation trajectory, its corresponding operation time can be obtained.

[0150] If, based on the operation time and the effective time of the game element's determination area, it is determined that the operation trajectory falls within the determination area within the effective time of the determination area, a first game feedback mechanism and / or a first game scoring mechanism may be triggered; if, based on the operation time and the effective time of the game element's determination area, it is determined that the operation trajectory does not fall within the determination area within the effective time of the determination area, a second game feedback mechanism and / or a second game scoring mechanism may be triggered. The game system continuously generates new game elements and monitors new operation trajectories, forming a complete dynamic interactive loop.

[0151] The aforementioned first / second game feedback mechanism and first / second game scoring mechanism can be flexibly configured by the developer. For example, upon a successful / failed hit, corresponding animations, sound effects, vibrations, and other game feedback mechanisms can be triggered, and the game score can be calculated based on the hit result.

[0152] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0153] In one embodiment, based on the same inventive concept, see Figure 9 As shown, a human-computer interaction control device is also provided, comprising: Acquisition module 901, used to acquire the target user image; A first determining module 902 is configured to determine a human body interaction point of a target user from the target user image, and determine an adaptive interaction area from the target user image that includes the human body interaction point and is adapted to the body shape of the target user; A second determining module 903 is configured to determine a mapping point of the human body interaction point in the front-end interactive page according to a relative positional relationship between the human body interaction point and the adaptive interactive area; The control module 904 is configured to implement interactive control of the front-end interactive page based on the mapping point.

[0154] In one embodiment, the first determination module 902 is used to determine the first position information of the human body key points of the target user in the target user image; and determine an adaptive interaction area from the target user image that includes the human body interaction points and is adapted to the body shape of the target user based on the first position information.

[0155] In one embodiment, the first determination module 902 is used to determine a plurality of first candidate interaction areas that are adapted to the body shape of the target user from the target user image based on the first position information; obtain the second position information of the human body interaction point in the target user image; and determine the adaptive interaction area to which the human body interaction point belongs from the plurality of first candidate interaction areas based on the second position information.

[0156] In one embodiment, the human body key points include a first key point, and the multiple first candidate interaction areas include a top interaction area located between the first key point and the upper interaction boundary line, and a lower interaction area located between the first key point and the lower interaction boundary line.

[0157] In one embodiment, the human body key points also include a second key point located below the first key point, and the lower interaction area includes an intermediate interaction area between the first key point and the second key point and a bottom interaction area between the second key point and the interaction lower boundary line; the adaptive interaction area is one of the top interaction area, the intermediate interaction area and the bottom interaction area.

[0158] In one embodiment, the first determination module 902 is used to determine a target detection frame covering the target user from the target user image; determine the upper detection frame line of the target detection frame as the interaction upper boundary line; or, the human body key point includes a third key point; obtain the first height feature parameter of the target user; and determine the interaction upper boundary line based on the first height feature parameter and the first position information of the third key point.

[0159] In one embodiment, the human body key points include a fourth key point, and the first determining module 902 is configured to obtain a second height characteristic parameter of the target user; The interaction lower boundary line is determined according to the second height characteristic parameter and the first position information of the fourth key point.

[0160] In one embodiment, each of the first candidate interaction areas is located between two longitudinal boundary lines, and the first determination module 902 is used to determine a target detection frame covering the target user from the target user image; and determine the longitudinal detection frame line of the target detection frame as the longitudinal boundary line.

[0161] In one embodiment, the first determination module 902 is used to determine a target detection frame covering the human interaction point and the target user from the target user image; and determine an adaptive interaction area from the target detection frame that includes the human interaction point and is adapted to the body shape of the target user.

[0162] In one embodiment, the first determination module 902 is used to determine a plurality of second candidate interaction areas that are adapted to the body shape of the target user from the target detection frame; obtain the second position information of the human body interaction point in the target user image; and determine the adaptive interaction area to which the human body interaction point belongs from the plurality of second candidate interaction areas based on the second position information.

[0163] In one embodiment, the acquisition module 901 is used to, when the interaction request is an off-vehicle interaction request, perform image capture of the area outside the vehicle through the vehicle's off-vehicle image capture device to obtain the target user image; the front-end interaction page is a page projected and displayed through the vehicle's projection device; and / or, the acquisition module 901 is used to, when the interaction request is an in-vehicle interaction request, perform image capture of the in-vehicle area through the vehicle's in-vehicle image capture device to obtain the target user image; the front-end interaction page is a page displayed through the in-vehicle display device.

[0164] In one embodiment, the human-computer interaction control device also includes a verification module for obtaining the identity authentication information of the target user before determining an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user from the target user image; and determining that the identity authentication information matches the preset legal identity information.

[0165] In one embodiment, the second determination module 903 is used to obtain the relative position relationship between the reference point in the adaptive interaction area and the human body interaction point in the target user image; and determine the mapping point of the human body interaction point in the front-end interaction page based on the reference mapping point of the reference point in the front-end interaction page and the relative position relationship.

[0166] In one embodiment, the second determination module 903 is used to obtain the area size information of the adaptive interaction area; determine the mapping point of the human body interaction point in the front-end interaction page based on the reference mapping point of the reference point in the front-end interaction page, the relative position relationship and the area size information.

[0167] In one embodiment, the area size information includes the width of the adaptive interaction area. and height The second determining module 903 is used to determine the Calculate the horizontal coordinate of the mapping point of the human body interaction point in the front-end interaction page ;according to Calculate the vertical coordinate of the mapping point of the human body interaction point in the front-end interaction page ;in, represents the position coordinates of the reference mapping point in the front-end interactive page, represents the displacement of the human body interaction point relative to the reference point in the horizontal axis direction of the target user image, represents the displacement of the human body interaction point relative to the reference point in the vertical coordinate axis direction of the target user image, and They respectively represent the first configuration parameter and the second configuration parameter configured for the front-end interactive page.

[0168] In one embodiment, when multiple frames of the target user image are acquired, the control module 904 is configured to determine the target user's operation trajectory based on the mapping points corresponding to each target user image; and implement interactive control based on the operation trajectory.

[0169] In one embodiment, when the front-end interactive page is a game page, the control module 904 is used to determine whether the operation trajectory falls into the corresponding judgment area of the game element in the game page; and trigger the corresponding game feedback mechanism and / or game scoring mechanism according to the judgment result.

[0170] It should be understood that for the sake of brevity, the contents described in some embodiments will not be repeated in this embodiment.

[0171] In one embodiment, see Figure 10 As shown, an electronic device is provided, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program, and the processor 1001 executes the computer program. The processor executes the computer program to implement the steps of the method introduced above, which will not be repeated here.

[0172] Processor 1001 can be an integrated circuit chip with signal processing capabilities. Processor 1001 can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor.

[0173] The memory 1002 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read Only Memory), EPROM (Erasable Programmable Read-Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory).

[0174] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the application of the solution of the present application. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0175] In one embodiment, based on the same inventive concept, a computer-readable storage medium is also provided, such as a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash disk, an SD (Secure Digital) card, an MMC (Multi-Media Card) card, etc., in which one or more programs for implementing the above steps are stored. These one or more programs can be executed by one or more processors to implement the steps of the methods in the above embodiments, which will not be repeated here.

[0176] In one embodiment, based on the same inventive concept, a computer program product is further provided, including a computer program, wherein the computer program implements any of the above methods when executed by a processor.

[0177] The program code for executing the computer program product of the present application may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0178] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer-readable storage media according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] It should be noted that the diagrams provided in the present embodiment are only schematic illustrations of the basic concept of the present application. The diagrams only show the components related to the present application rather than the number, shape and size of the components when actually implemented. The type, quantity and ratio of each component can be changed at will during actual implementation, and the component layout pattern may also be more complicated. The structures, ratios, sizes, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read. They are not used to limit the restrictive conditions that can be implemented in this application. Therefore, they have no technical significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed in this application without affecting the effect and purpose that can be achieved by this application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description and are not used to limit the scope of the implementation of this application. The change or adjustment of their relative relationship should also be considered as the scope of the implementation of this application without substantial change in the technical content.

[0183] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places herein does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0184] As used herein, unless the context clearly indicates otherwise, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include additional steps or elements.

[0185] The definition of inclusion herein, such as the terms “having”, “may have”, “include” or “may include” as used herein, indicates the existence of the corresponding functions, operations, elements, etc. herein, and does not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms “including” or “having” as used herein indicate the existence of the features, numbers, steps, operations, elements, components or their combination described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, components or their combination.

[0186] In the embodiments of the present application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present application does not constitute a restriction on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary restriction. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0187] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A human-computer interaction control method, characterized in that: include: Upon receiving an interaction request, obtaining a target user image; determining a human body interaction point of a target user from the target user image, and determining an adaptive interaction area from the target user image that includes the human body interaction point and is adapted to the body shape of the target user; Determining a mapping point of the human body interaction point in the front-end interaction page according to a relative positional relationship between the human body interaction point and the adaptive interaction area; Interactive control of the front-end interactive page is achieved based on the mapping point.

2. The human-computer interaction control method according to claim 1, wherein: Determining, from the target user image, an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user, comprising: Determining first position information of a key point of a human body of the target user in the target user image; An adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user is determined from the target user image according to the first position information.

3. The human-computer interaction control method according to claim 2, wherein: Determining, from the target user image according to the first position information, an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user, including: determining, from the target user image according to the first position information, a plurality of first candidate interaction areas that are adapted to the body shape of the target user; Acquire second position information of the human body interaction point in the target user image; An adaptive interaction area to which the human body interaction point belongs is determined from the multiple first candidate interaction areas according to the second position information.

4. The human-computer interaction control method according to claim 3, wherein: The human body key points include a first key point, and the multiple first candidate interaction areas include a top interaction area located between the first key point and an upper interaction boundary line, and a lower interaction area located between the first key point and a lower interaction boundary line.

5. The human-computer interaction control method according to claim 4, wherein: The human body key points also include a second key point located below the first key point, and the lower interaction area includes an intermediate interaction area between the first key point and the second key point and a bottom interaction area between the second key point and the interaction lower boundary line; the adaptive interaction area is one of the top interaction area, the intermediate interaction area and the bottom interaction area.

6. The human-computer interaction control method according to claim 4, wherein: The method for determining the upper boundary line of interaction includes: Determining a target detection frame covering the target user from the target user image; and determining an upper detection frame line of the target detection frame as the upper interaction boundary line; or, The human body key points include a third key point; obtaining a first height characteristic parameter of the target user; and determining the upper interaction boundary line according to the first height characteristic parameter and the first position information of the third key point.

7. The human-computer interaction control method according to claim 4, wherein: The human body key points include a fourth key point, and the method for determining the interaction lower boundary line includes: Obtaining a second height characteristic parameter of the target user; The interaction lower boundary line is determined according to the second height characteristic parameter and the first position information of the fourth key point.

8. The human-computer interaction control method according to claim 4, wherein: Each of the first candidate interaction areas is located between two longitudinal boundary lines, and the longitudinal boundary lines are determined in the following manner: Determining a target detection frame covering the target user from the target user image; A vertical detection frame line of the target detection frame is determined as the vertical boundary line.

9. The human-computer interaction control method according to claim 1, wherein: Determining, from the target user image, an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user, comprising: Determining a target detection frame covering the human interaction point and the target user from the target user image; An adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user is determined from the target detection frame.

10. The human-computer interaction control method according to claim 9, wherein: Determining an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user from the target detection frame includes: Determining a plurality of second candidate interaction areas that are adapted to the body shape of the target user from the target detection frame; Acquire second position information of the human body interaction point in the target user image; An adaptive interaction area to which the human body interaction point belongs is determined from a plurality of second candidate interaction areas according to the second position information.

11. The human-computer interaction control method according to claim 1, wherein: The step of obtaining the target user image includes: When the interaction request is an off-vehicle interaction request, the target user image is obtained by capturing an image of an area outside the vehicle through an off-vehicle image capture device of the vehicle; the front-end interaction page is a page projected and displayed by a vehicle projection device; and / or, When the interaction request is an in-vehicle interaction request, the target user image is obtained by capturing the image of the in-vehicle area through the in-vehicle image capture device of the vehicle; the front-end interaction page is a page displayed by the in-vehicle display device.

12. The human-computer interaction control method according to claim 1, wherein: Before determining, from the target user image, an adaptive interaction area that includes the human body interaction point and is adapted to the body shape of the target user, the method includes: Obtaining identity authentication information of the target user; Determine whether the identity verification information matches the preset legal identity information.

13. The human-computer interaction control method according to any one of claims 1 to 12, characterized in that: Determining a mapping point of the human body interaction point in the front-end interactive page according to a relative positional relationship between the human body interaction point of the target user and the adaptive interactive area includes: Acquire a relative positional relationship between a reference point in the adaptive interaction area and the human body interaction point in the target user image; The mapping point of the human body interaction point in the front-end interaction page is determined according to the reference mapping point of the reference point in the front-end interaction page and the relative position relationship.

14. The human-computer interaction control method according to claim 13, wherein: Determining a mapping point of the human body interaction point in the front-end interaction page according to a reference mapping point of the reference point in the front-end interaction page and the relative position relationship includes: Acquiring area size information of the adaptive interaction area; The mapping point of the human body interaction point in the front-end interaction page is determined according to the reference mapping point of the reference point in the front-end interaction page, the relative position relationship and the area size information.

15. The human-computer interaction control method according to claim 14, wherein: The area size information includes the width of the adaptive interaction area and height , determining the mapping point of the human body interaction point in the front-end interactive page according to the reference mapping point of the reference point in the front-end interactive page, the relative position relationship, and the area size information, including: according to Calculate the horizontal coordinate of the mapping point of the human body interaction point in the front-end interaction page ; according to Calculate the vertical coordinate of the mapping point of the human body interaction point in the front-end interaction page ; in, represents the position coordinates of the reference mapping point in the front-end interactive page, represents the displacement of the human body interaction point relative to the reference point in the horizontal axis direction of the target user image, represents the displacement of the human body interaction point relative to the reference point in the vertical coordinate axis direction of the target user image, and They respectively represent the first configuration parameter and the second configuration parameter configured for the front-end interactive page.

16. The human-computer interaction control method according to any one of claims 1 to 12, characterized in that: When multiple frames of the target user image are acquired, interactive control of the front-end interactive page is implemented based on the mapping points, including: determining an operation trajectory of the target user based on mapping points corresponding to each of the target user images; Interactive control is achieved based on the operation trajectory.

17. The human-computer interaction control method according to claim 16, wherein: When the front-end interactive page is a game page, implementing interactive control based on the operation trajectory includes: Determining whether the operation track falls into a corresponding determination area of a game element in the game page; The corresponding game feedback mechanism and / or game scoring mechanism is triggered according to the judgment result.

18. A human-computer interaction control device, characterized in that: The device comprises: An acquisition module, used to acquire the target user's image; a first determining module, configured to determine a human body interaction point of a target user from the target user image, and determine an adaptive interaction area from the target user image that includes the human body interaction point and is adapted to the body shape of the target user; A second determining module is configured to determine a mapping point of the human body interaction point in the front-end interactive page according to a relative positional relationship between the human body interaction point and the adaptive interactive area; A control module is used to implement interactive control of the front-end interactive page based on the mapping point.

19. An electronic device, characterized in that: The system comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 17 is implemented.

Images