A Human-Computer Interaction Method and Device Based on a Knowledge Graph of Binocular Cameras

Abstract

The human-computer interaction method, device and storage medium of the knowledge graph based on a binocular camera proposed in this application obtain interaction information of a preset number of frames N based on the binocular camera, determine corresponding manipulation information based on the interaction information, and the manipulation information includes an attitude type, valid information, and control parameters. The valid information is used to indicate whether to perform an interaction. If the valid information indicates an interaction, an interface corresponding to the manipulation information is matched based on the attitude type, and the control parameters are sent to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameters. Thus, this application manipulates the knowledge graph through the obtained interaction information so that the knowledge graph performs corresponding display, thereby enabling interaction with the knowledge graph without any additional access devices, facilitating the user's manipulation process and saving costs.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and particularly to a human-computer interaction method, device, and storage medium based on a knowledge graph of a binocular camera. Background Art

[0002] Knowledge graphs have the characteristics of a large number of concepts and complex data relationships. Based on these characteristics, knowledge graphs are usually displayed in a three-dimensional space to comprehensively display the content of the knowledge graph in a visualization interface. Therefore, operators need to operate on the visualization interface of the knowledge graph in a three-dimensional space.

[0003] In related technologies, operators use additional devices (such as a mouse, wireless projection pen) to manipulate the knowledge graph, and the knowledge graph needs to be adapted to different additional devices, making the manipulation process relatively complex. Summary of the Invention

[0004] This application provides a human-computer interaction method, device, and storage medium based on a knowledge graph of a binocular camera to solve the problem of relatively complex manipulation in the above method.

[0005] A first aspect embodiment of this application proposes a human-computer interaction method based on a knowledge graph of a binocular camera, and the method includes:

[0006] Obtain interaction information of a preset number of frames N based on a binocular camera, where N is a positive integer and N≥2;

[0007] Determine corresponding manipulation information based on the interaction information, where the manipulation information includes an attitude type, valid information, and control parameters, and the valid information is used to indicate whether to perform an interaction;

[0008] If the valid information indicates an interaction, match an interface corresponding to the manipulation information based on the attitude type, where there is a mapping relationship between the attitude type and the interface;

[0009] Send the control parameters to the knowledge graph by calling the interface so that the knowledge graph performs corresponding display based on the interface and the control parameters.

[0010] A second aspect embodiment of this application proposes a human-computer interaction device based on a knowledge graph of a binocular camera, including:

[0011] An acquisition module, configured to obtain interaction information of a preset number of frames N based on a binocular camera;

[0012] An analysis module, configured to determine corresponding manipulation information based on the interaction information, where the manipulation information includes an attitude type, valid information, and control parameters, and the valid information is used to indicate whether to perform an interaction;

[0013] A comprehensive judgment module, configured to, if the valid information indicates an interaction, match an interface corresponding to the control information based on the posture type, where there is a mapping relationship between the posture type and the interface;

[0014] A sending module, configured to send the control parameters to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameters.

[0015] The computer storage medium provided in the third aspect embodiment of the present application, where the computer storage medium stores computer-executable instructions; after being executed by a processor, the computer-executable instructions can implement the method described in the first aspect above.

[0016] The technical solution provided in the embodiment of the present application at least brings the following beneficial effects:

[0017] The human-computer interaction method, device and storage medium of the knowledge graph based on a binocular camera proposed in the present application obtain interaction information of a preset number of frames N based on the binocular camera, determine corresponding control information based on the interaction information, and the control information includes a posture type, valid information and control parameters. The valid information is used to indicate whether to perform an interaction. If the valid information indicates an interaction, an interface corresponding to the control information is matched based on the posture type, and the control parameters are sent to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameters. It can be seen that the present application manipulates the knowledge graph through the obtained interaction information so that the knowledge graph performs corresponding display, so that the user can interact with the knowledge graph without any additional access device, which facilitates the user's manipulation process and saves costs.

[0018] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0020] Figure 1 is a schematic flowchart of a human-computer interaction method of a knowledge graph based on a binocular camera according to an embodiment of the present application;

[0021] Figure 2 is a schematic structural diagram of a human-computer interaction device of a knowledge graph based on a binocular camera according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where like or similar reference numerals denote like or similar elements or elements having like or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application.

[0023] The human-computer interaction method and device based on the knowledge graph of a binocular camera according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0024] Embodiment 1

[0025] Figure 1 FIG. is a schematic flowchart of a human-computer interaction method based on the knowledge graph of a binocular camera according to an embodiment of the present application. As Figure 1 shown, it may include:

[0026] Step 101: Obtain interaction information of a preset number of frames N based on the binocular camera.

[0027] Wherein, in the embodiments of the present application, the above-mentioned interaction information may include hands position information, hands movement information, and other information, and the other information is information of other body parts except the hands (for example, facial information). The valid information may include first valid information and second valid information. The first valid information is used to indicate whether the hands position information in the above-mentioned interaction information is valid, and the second valid information indicates whether the above-mentioned interaction information is valid.

[0028] In addition, in an embodiment of the present application, the above-mentioned interaction information will be updated in the form of a sliding window. Specifically, a queue length M may be preset, and then the most recent M frames of interaction information sent by the binocular camera are saved in the queue.

[0029] Furthermore, in an embodiment of the present application, when interaction information of a preset number of frames N is obtained based on the binocular camera, the interaction information is analyzed, where N is a positive integer and N≥2, and N<M (for example, N = M / 2).

[0030] Step 102: Determine corresponding manipulation information based on the interaction information. The manipulation information includes posture type, valid information, and control parameters.

[0031] Wherein, in an embodiment of the present application, the valid information may be used to indicate whether interaction is performed.

[0032] In addition, in an embodiment of the present application, the method for determining corresponding manipulation information based on the interaction information may include the following steps:

[0033] Step 1021: Determine a corresponding motion vector and first valid information based on the hands position information.

[0034] Among them, in an embodiment of the present application, the hand position information may include the hand coordinates corresponding to each frame in a preset number of frames, and the hand coordinates may include the left - hand coordinates and / or the right - hand coordinates.

[0035] In addition, in an embodiment of the present application, the method for determining the corresponding motion vector and the first valid information based on the hand position information may include the following steps:

[0036] Step a: Determine the motion vector based on the hand coordinates of the first frame and the last frame in the preset number of frames N.

[0037] Among them, in an embodiment of the present application, the difference between the hand coordinates of the first frame and the last frame may be taken as the motion vector corresponding to the hand position information.

[0038] For example, in an embodiment of the present application, assuming that the hand position information is the left - hand coordinates, and the left - hand coordinate of the first frame is (1, 3) and the left - hand coordinate of the last frame is (2, 5), then the difference between these two coordinates is taken, and the left - hand motion vector corresponding to the hand position information is calculated as (1, 2).

[0039] Step b: Calculate the corresponding N - 1 interval sub - motion vectors based on the hand coordinates between every two consecutive frames in the preset number of frames N.

[0040] Among them, in an embodiment of the present application, the difference between the hand coordinates between every two consecutive frames may be taken as the corresponding interval sub - motion vector.

[0041] For example, assuming N = 5, the 4 interval sub - motion vectors corresponding to the hand position information calculated through the above steps are: (1, 2), (-1, 3), (-9, 0), (5, 10), and the motion vector calculated through the first frame and the last frame is (6, 12).

[0042] Step c: Calculate the corresponding N - 1 first included angles between each interval sub - motion vector and the motion vector based on each interval sub - motion vector and the motion vector.

[0043] Among them, in an embodiment of the present application, the corresponding first included angle between each interval sub - motion vector and the motion vector can be calculated based on the operation between vectors.

[0044] Step d: Determine the number of the first included angles less than the first included - angle threshold among the N - 1 first included angles.

[0045] For example, the first included - angle threshold may be 30 degrees.

[0046] Step e: If the quantity is greater than the first threshold, determine that the first valid information indicates that the hands position information is valid.

[0047] Among them, in an embodiment of the present application, if the quantity is greater than the first threshold, it indicates that the user's hands have made a straight movement. At this time, determine that the hands position information is valid, and based on this, determine that the first valid information indicates that the hands position information is valid; otherwise, it is considered that the user's hands have made frequent jitters and are not intended to manipulate the knowledge graph, then determine that the hands position information is invalid, and at this time, the first valid information indicates that the hands position information is invalid.

[0048] Exemplarily, assume that the first threshold is n, and n = 70% of the preset number of frames N. Then, when the number of the first included angles less than 30 degrees among N - 1 first included angles is greater than n, it is determined that the obtained hands position information is valid; otherwise, it is determined that the obtained hands position information is invalid.

[0049] It should be noted that through the above steps, it can be determined whether the hands position information obtained by the binocular camera is valid, so as to determine whether the user intends to manipulate the knowledge graph, and further, the unconscious movements of the user's hands can be eliminated.

[0050] Step 1022: If the first valid information indicates that the motion vector is valid, determine the control parameter according to the motion vector and the default parameter.

[0051] Among them, in an embodiment of the present application, the default parameter may include a manipulation mode and a motion distance parameter. And, in an embodiment of the present application, the manipulation mode may include two different manipulation modes, namely fixed - distance manipulation and following the motion vector of the hand.

[0052] Specifically, in an embodiment of the present application, the fixed - distance manipulation mode will move a corresponding distance in the direction pointed by the motion vector according to the value set by the motion distance parameter; the following - the - motion - vector - of - the - hand manipulation mode will perform corresponding operations according to the vector obtained by multiplying the motion vector by the motion distance parameter.

[0053] Step 1023: Determine the corresponding gesture type based on the hands motion information.

[0054] Among them, in an embodiment of the present application, the above hands motion information may include the hands gesture type corresponding to each frame in the preset number of frames, and the hands gesture type may include the left - hand gesture type and / or the right - hand gesture type. Specifically, the left - hand gesture type may include fist - clenching and opening, and the right - hand gesture type may include fist - clenching and opening.

[0055] Moreover, in one embodiment of the present application, the method for determining the corresponding gesture type based on the two-handed action information may include: determining the gesture type with the largest same quantity in the preset number of frames as the gesture type corresponding to the two-handed action information.

[0056] Exemplarily, it is assumed that the two-handed action information includes the left-hand gesture type, and the left-hand gesture type with the largest same quantity in the preset number of frames is a fist. Then, the gesture type determined based on the two-handed action information is the left hand making a fist.

[0057] Step 1024: Determine the second valid information based on other information.

[0058] Among them, in one embodiment of the present application, the other information may include the facial direction corresponding to each frame in the preset number of frames.

[0059] Moreover, in one embodiment of the present application, the method for determining the second valid information based on other information may include the following steps:

[0060] Step 1: Calculate N second included angles between the facial direction of each frame in the preset number of frames N and the vertical direction of the binocular camera.

[0061] Among them, in one embodiment of the present application, the second included angle between the facial direction of each frame and the vertical direction of the binocular camera may be calculated based on the operation between vectors.

[0062] Step 2: Determine the quantity of the N second included angles that are greater than the second included angle threshold.

[0063] Exemplarily, the second included angle threshold may be 30 degrees.

[0064] Step 3: If the quantity is greater than the second threshold, determine that the second valid information indicates that the interaction position information is invalid.

[0065] Among them, in one embodiment of the present application, if the quantity is greater than the second threshold, it indicates that the user's face is not facing the direction of the camera. Based on this, it is considered that the user does not want to manipulate the knowledge graph, and then determine that the second valid information indicates that the interaction information is invalid; otherwise, it indicates that the user's face is facing the direction of the camera. Based on this, it is considered that the user wants to manipulate the knowledge graph, and then determine that the second valid information indicates that the interaction information is valid.

[0066] Step 1025: Determine the valid information based on the first valid information and the second valid information.

[0067] Among them, in an embodiment of the present application, the method for determining valid information based on the first valid information and the second valid information may include: if the first valid information indicates that the hand coordinate information is valid and the second valid information indicates that the interaction information is valid, determine that the valid information indicates an interaction; otherwise, determine that the valid information indicates no interaction.

[0068] Step 103, if the valid information indicates an interaction, match the interface corresponding to the manipulation information based on the posture type.

[0069] Among them, in an embodiment of the present application, there is a mapping relationship between the posture type and acceptance, and the posture type corresponds to the interface one by one.

[0070] Exemplarily, in an embodiment of the present application, assuming that the posture type is a left - hand fist, the interface corresponding to the manipulation information matched based on the posture type is the left - hand fist interface.

[0071] Step 104, send the control parameters to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the manipulation parameters.

[0072] Among them, in an embodiment of the present application, after the knowledge graph receives the control parameters through the interface, it can determine the control type to be performed on the graph based on the interface. There is a mapping relationship between the interface and the control type, and the interface corresponds to the control type one by one. Exemplarily, assuming that the knowledge graph receives the control parameters through the left - hand fist interface, the control type can be determined as graph translation through the mapping relationship between the interface and the control type; assuming that the knowledge graph receives the control parameters through the left - hand open interface, the control type can be determined as graph rotation through the mapping relationship between the interface and the control type.

[0073] And, in an embodiment of the present application, after the knowledge graph determines the control type, it can perform corresponding control on the knowledge graph through the control parameters, as shown in Table 1.

[0074] Table 1

[0075]

[0076] Referring to Table 1, the control type realizes the corresponding control effect on the knowledge graph according to the control parameters.

[0077] Exemplarily, in an embodiment of the present application, assuming that the control type is graph translation and the manipulation mode in the control parameters is the fixed - example manipulation mode, the knowledge graph will determine the distance according to the movement distance parameter, and then move that distance in the direction pointed by the movement vector to achieve the translation of the three - dimensional knowledge graph on the plane perpendicular to the camera.

[0078] Also, in an embodiment of the present application, when the control type is map zooming, the map can be manipulated only according to the magnification / reduction ratio, without the need for movement, and at this time, the value of the control parameter is not required.

[0079] The human-computer interaction method for a knowledge graph based on a binocular camera proposed in the present application obtains interaction information of a preset number of frames N based on the binocular camera, determines corresponding manipulation information based on the interaction information, and the manipulation information includes a posture type, valid information, and a control parameter. The valid information is used to indicate whether to perform an interaction. If the valid information indicates an interaction, an interface corresponding to the manipulation information is matched based on the posture type, and the control parameter is sent to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameter. It can be seen that the present application manipulates the knowledge graph through the obtained interaction information to enable the knowledge graph to perform corresponding display, so that the user can interact with the knowledge graph without any additional access device, which facilitates the user's manipulation process and saves costs.

[0080] Embodiment 2

[0081] Figure 2 FIG. is a schematic structural diagram of a human-computer interaction device for a knowledge graph based on a binocular camera according to an embodiment of the present application, as Figure 2 shown, and may include:

[0082] An acquisition module 201, configured to obtain interaction information of a preset number of frames N based on the binocular camera;

[0083] An analysis module 202, configured to determine corresponding manipulation information based on the interaction information, where the manipulation information includes a posture type, valid information, and a control parameter, and the valid information is used to indicate whether to perform an interaction;

[0084] A comprehensive judgment module 203, configured to, if the valid information indicates an interaction, match an interface corresponding to the manipulation information based on the posture type, where there is a mapping relationship between the posture type and the interface;

[0085] A sending module 204, configured to send the control parameter to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameter.

[0086] Among them, in an embodiment of the present application, the above analysis module 202 includes:

[0087] A two-handed coordinate analysis module 2021, configured to determine a corresponding motion vector and first valid information based on the two-handed position information, and if the first valid information indicates that the motion vector is valid, determine the control parameter according to the motion vector and the default parameter;

[0088] The two - hand motion analysis module 2022 is used to determine the corresponding posture type based on the two - hand motion information;

[0089] The other - information analysis module 2023 is used to determine the second valid information based on other information;

[0090] The valid - information determination module 2024 is used to determine the valid information based on the first valid information and the second valid information.

[0091] The human - computer interaction device based on the knowledge graph of binocular cameras proposed in this application obtains interaction information of a preset number of frames N based on the binocular cameras, determines the corresponding control information based on the interaction information. The control information includes the posture type, the valid information, and the control parameters. The valid information is used to indicate whether to perform an interaction. If the valid information indicates to perform an interaction, the interface corresponding to the control information is matched based on the posture type, and the control parameters are sent to the knowledge graph by calling the interface, so that the knowledge graph performs the corresponding display based on the interface and the control parameters. It can be seen that this application manipulates the knowledge graph through the obtained interaction information to make the knowledge graph perform the corresponding display, so that the user can interact with the knowledge graph without any additional access devices, which facilitates the user's control process and saves costs.

[0092] To implement the above - mentioned embodiments, the present disclosure also proposes a computer storage medium.

[0093] The computer storage medium provided by the embodiments of the present disclosure stores an executable program; after being executed by a processor, the executable program can implement the method as Figure 1 shown in any one.

[0094] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above - mentioned terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0095] Any process or method description, whether in a flowchart or otherwise described herein, can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process. The scope of the preferred embodiments of the present application includes additional implementations where functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed. This should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0096] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A human-computer interaction method based on a knowledge graph of a binocular camera, characterized in that, The method includes: Obtaining interaction information of a preset number of frames N based on a binocular camera, where N is a positive integer and N≥2; Determining corresponding manipulation information based on the interaction information, the manipulation information including an attitude type, valid information, and control parameters, the valid information being used to indicate whether an interaction is performed; If the valid information indicates that an interaction is performed, matching an interface corresponding to the manipulation information based on the attitude type, where there is a mapping relationship between the attitude type and the interface; Sending the control parameters to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameters; Wherein, the interaction information includes hands position information, hands motion information, and other information, the other information being information of other body parts except the hands, the valid information includes first valid information and second valid information, the first valid information being used to indicate whether the hands position information is valid, and the second valid information indicating whether the interaction information is valid; The determining corresponding manipulation information based on the interaction information includes: Determining a corresponding motion vector and the first valid information based on the hands position information, if the first valid information indicates that the motion vector is valid, determining control parameters according to the motion vector and default parameters; Determining a corresponding attitude type based on the hands motion information; Determining the second valid information based on the other information, where the other information includes the face direction corresponding to each frame in the preset number of frames N, calculating N second included angles between the face direction of each frame in the preset number of frames N and the vertical direction of the binocular camera, determining the number of the N second included angles greater than a second included angle threshold, if the number is greater than a second threshold, determining that the second valid information indicates that the interaction position information is invalid; Determining the valid information based on the first valid information and the second valid information.

2. The method according to claim 1, wherein The hands position information includes the hands coordinates corresponding to each frame in the preset number of frames, the hands coordinates including the left hand coordinates and / or the right hand coordinates, The determining the corresponding motion vector and the first valid information based on the hands position information includes: Determining a motion vector based on the hands coordinates of the first frame and the last frame in the preset number of frames N; Calculating corresponding N - 1 interval sub - motion vectors based on the hands coordinates between every two consecutive frames in the preset number of frames N; Calculating corresponding N - 1 first included angles between each interval sub - motion vector and the motion vector based on each interval sub - motion vector and the motion vector; Determining the number of the N - 1 first included angles less than a first included angle threshold; If the number is greater than a first threshold, determining that the first valid information indicates that the hands position information is valid.

3. The method according to claim 1, wherein The hands motion information includes the hands attitude types corresponding to each frame in the preset number of frames, the hands attitude types including the left hand attitude type and / or the right hand attitude type, The determining the corresponding attitude type based on the hands motion information includes: Determining the attitude type corresponding to the hands motion information as the hands attitude type with the most occurrences of the same number in the preset number of frames.

4. The method according to claim 1, wherein Determining the effective information based on the first effective information and the second effective information includes: If the first effective information indicates that the two - hand coordinate information is effective and the second effective information indicates that the interaction information is effective, determine that the effective information indicates an interaction.

5. The method according to claim 1, wherein The method further includes: If the effective information indicates no interaction, no control parameters will be sent to the knowledge graph.

6. A human-computer interaction system based on a knowledge graph of a binocular camera, characterized in that, The system includes: An acquisition module, configured to acquire interaction information of a preset number of frames N based on a binocular camera; An analysis module, configured to determine corresponding manipulation information based on the interaction information, where the manipulation information includes a posture type, effective information, and control parameters, and the effective information is used to indicate whether an interaction is performed; A comprehensive judgment module, configured to, if the effective information indicates an interaction, match an interface corresponding to the manipulation information based on the posture type, where there is a mapping relationship between the posture type and the interface; A sending module, configured to send the control parameters to the knowledge graph by calling the interface, so that the knowledge graph performs corresponding display based on the interface and the control parameters; Wherein, the interaction information includes two - hand position information, two - hand motion information, and other information, the other information is information of other body parts except the two hands, the effective information includes first effective information and second effective information, the first effective information is used to indicate whether the two - hand position information is effective, and the second effective information indicates whether the interaction information is effective; The analysis module includes: A two - hand coordinate analysis module, configured to determine a corresponding motion vector and the first effective information based on the two - hand position information. If the first effective information indicates that the motion vector is effective, determine the control parameters according to the motion vector and default parameters; A two - hand motion analysis module, configured to determine a corresponding posture type based on the two - hand motion information; An other - information analysis module, configured to determine the second effective information based on the other information. The other information includes the facial direction corresponding to each frame in the preset number of frames N, calculate N second included angles between the facial direction of each frame in the preset number of frames N and the vertical direction of the binocular camera, determine the number of the N second included angles greater than a second included - angle threshold, and if the number is greater than a second threshold, determine that the second effective information indicates that the interaction position information is invalid; An effective - information determination module, configured to determine the effective information based on the first effective information and the second effective information.

Images