Multi-user collaborative design interaction main body switching method and device

Abstract

The invention discloses a multi-user collaborative design interaction main body switching method and device, and relates to the technical field of multi-user collaborative design interaction, the method is applied to a man-machine interaction system supporting a multi-user collaborative design scene, and the method comprises the following steps: determining all users participating in the multi-user collaborative design scene at the current moment based on a current image, the posture grade, the displacement and the speed of each user at the current moment are obtained; based on the posture level, displacement and speed of the same user at the current moment, determining the interaction strength of the corresponding user at the current moment; and based on the interaction strength of all the users participating in the multi-user collaborative design scene at the current moment, determining an interaction main body at the current moment, and controlling to switch the interaction main body. And the rationality of automatic switching of the interaction main bodies of the man-machine interaction system in a multi-person collaborative design scene is improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-person collaborative design interaction, and in particular to a method and device for switching subjects of multi-person collaborative design interaction. Background Art

[0002] With the continuous development of the machinery industry, the rapid increase in the complexity of mechanical products has brought new challenges to engineering design problems. Complex product design is usually a process with a multi-level structure and contains many system goals and nonlinear constraints, showing the characteristics of multi-disciplinary, multi-variable, multi-coupling, multi-agent and high-order. However, the existing design methods are discrete and interdisciplinary and isolated, without an integrated design environment and interoperability of real-time data, resulting in inefficient design processes, long design cycles, high costs, and low quality, which do not meet the design requirements of complex products.

[0003] In recent years, with the development of human-computer interaction technology, multi-person collaborative design environments that combine multi-modal natural interactions such as gestures and eye movements have become an important development trend in the future. The characteristics of multi-person collaborative design environments are multi-user participation, multi-disciplinary intersections, and visualization. This advanced design environment makes product design effects more intuitive and communication between designers smoother. A good multi-person collaborative design environment can meet the requirements of high integration, efficient communication, and rapid iteration of complex product design, thereby improving the communication efficiency and design efficiency of the design team. The multi-person collaborative design environment enables designers to work together in integrated product development and plays an important auxiliary role in the design process of complex products. However, the intensity of user interaction in the multi-person collaborative design environment is difficult to reflect, and the subject switching is inaccurate, which seriously affects the collaborative work of designers in the multi-person collaborative design environment. Summary of the invention

[0004] The purpose of this application is to provide a method and device for switching interactive subjects in multi-person collaborative design, which can improve the rationality of automatic switching of interactive subjects in a human-computer interaction system in a multi-person collaborative design scenario.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for switching a subject of a multi-person collaborative design interaction, wherein the method is applied to a human-computer interaction system supporting a multi-person collaborative design scenario, wherein the human-computer interaction system supporting a multi-person collaborative design scenario includes a depth camera;

[0007] The depth camera is used to obtain a multi-person collaborative design scene image sequence at a preset frequency;

[0008] The multi-person collaborative design interaction subject switching method comprises:

[0009] Acquire the multi-person collaborative design scene image at the current moment as the current image;

[0010] Determine all users participating in the multi-person collaborative design scene at the current moment based on the current image, as well as the posture level, displacement and speed of each user at the current moment;

[0011] Based on the posture level, displacement and speed of the same user at the current moment, determine the interaction intensity of the corresponding user at the current moment;

[0012] Based on the interaction intensity of all users participating in the multi-person collaborative design scenario at the current moment, determine the interaction subject at the current moment;

[0013] When the interaction subject at the current moment is different from the interaction subject at the previous moment, the interaction subject is switched.

[0014] Optionally, all users participating in the multi-person collaborative design scene at the current moment, as well as the posture level, displacement and speed of each user at the current moment are determined based on the current image, including:

[0015] Extracting first posture data of each user in the current image using the Openpose tool; the first posture data includes coordinates of multiple key points of the same user in a camera coordinate system;

[0016] The first posture data of all users are converted into the world coordinate system to obtain the second posture data corresponding to each user;

[0017] Determine any second body shape data as current body shape data;

[0018] Determine that the user corresponding to the current body posture data is the current user;

[0019] Based on the current body posture data, determining the displacement and speed of the current user at the current moment;

[0020] Determine the occlusion degree of the current user according to the interval where the number of key points in the current body posture data is located; the interval corresponds to the occlusion degree one by one; the occlusion degree includes an unrecognizable occlusion degree and multiple recognizable occlusion degrees;

[0021] When the occlusion degree of the current user is an unrecognizable occlusion degree, the current body posture data is updated and the process returns to the step of “determining that the user corresponding to the current body posture data is the current user”;

[0022] When the occlusion degree of the current user is a recognizable occlusion degree, determining that the individual posture level recognition model corresponding to the occlusion degree of the current user is the current individual posture level recognition model; the recognizable occlusion degree corresponds to the individual posture level recognition model one-to-one; the individual posture level recognition model is obtained by training a convolutional neural network using a plurality of second body posture data corresponding to the recognizable occlusion degree;

[0023] Inputting the current body posture data into a current individual posture grade recognition model to obtain the posture grade of the current user at the current moment;

[0024] Update the current posture data and return to the step of "determining that the user corresponding to the current posture data is the current user" until all the second posture data corresponding to the current image are traversed to obtain the posture level of each user at the current moment.

[0025] Optionally, after determining that the user corresponding to the current body posture data is the current user, the method further includes:

[0026] Using a continuity detection algorithm, it is determined whether the current user exists in the multi-person collaborative design scene image at the previous moment, and a first determination result is obtained;

[0027] If the first judgment result is no, the current user is determined to be a new user, and the user number of the current user is determined to be the maximum user number at the previous moment plus 1;

[0028] If the first judgment result is yes, it is determined that the current user is an existing user, and the user number of the current user at the previous moment is used.

[0029] Optionally, before obtaining the multi-person collaborative design scene image at the current moment as the current image, the method further includes:

[0030] Constructing a plurality of convolutional neural networks; the number of the convolutional neural networks is equal to the number of the recognizable occlusion degrees;

[0031] Determine any recognizable occlusion degree as the current recognizable occlusion degree;

[0032] Acquire a plurality of second body posture data of currently recognizable occlusion degree;

[0033] Annotating posture levels corresponding to a plurality of second posture data of currently identifiable occlusion degrees;

[0034] Taking multiple second body posture data of the currently recognizable occlusion degree as input and taking the posture level as output, the convolutional neural network is trained to obtain an individual posture level recognition model corresponding to the currently recognizable occlusion degree;

[0035] Update the current recognizable occlusion degree, and return to the step of "obtaining multiple second body posture data of the current recognizable occlusion degree" until all recognizable occlusion degrees are traversed to obtain the individual posture level recognition model corresponding to each recognizable occlusion degree.

[0036] Optionally, the convolutional neural network includes a convolutional layer, a maximum pooling layer, a flattening layer, a first fully connected layer and a second fully connected layer connected in sequence.

[0037] Optionally, based on the current body posture data, determining the displacement and speed of the current user at the current moment includes:

[0038] When the current user is a new user, update the current body shape data and return to the step of "determining that the user corresponding to the current body shape data is the current user";

[0039] When the current user is an existing user, obtaining the second posture data of the current user in the multi-person collaborative design scene image at a first time interval before the current moment as the historical second posture data;

[0040] Determine the coordinate mean of all key points in the historical second body data in the world coordinate system as the historical position of the current user;

[0041] Determine the coordinate mean of all key points in the current body data in the world coordinate system as the current position of the current user history;

[0042] Determine the displacement of the current user at the current moment based on the historical location and current location of the current user;

[0043] The ratio of the displacement of the current user at the current moment to the first time interval is determined as the speed of the current user at the current moment.

[0044] Optionally, the interaction strength is:

[0045]

[0046] Among them, L i，t represents the interaction intensity of the i-th user participating in the multi-person collaborative design scenario at time t; G i represents the posture level of the i-th user participating in the multi-person collaborative design scenario at time t; * represents multiplication operation; e represents a natural constant; ΔP i represents the displacement of the i-th user participating in the multi-person collaborative design scenario at time t; V i Represents the speed of the i-th user participating in the multi-person collaborative design scenario at time t.

[0047] Optionally, based on the interaction strength of all users participating in the multi-person collaborative design scenario at the current moment, determining the interaction subject at the current moment includes:

[0048] Determine any user as the current user;

[0049] When the current user is a new user, the interaction intensity of the current user at the current moment is used as an element to construct the interaction intensity sequence of the current user at the current moment;

[0050] When the current user is an existing user, obtain the interaction intensity sequence of the current user at the previous moment;

[0051] Add the interaction strength of the current user at the current moment as the last element to the interaction strength sequence of the current user at the previous moment to obtain the interaction strength sequence of the current user at the current moment;

[0052] Determine whether the number of elements in the interaction intensity sequence of the current user at the current moment reaches a preset number, and obtain a second determination result;

[0053] If the second judgment result is no, the current user is updated, and the process returns to the step of "when the current user is a new user, taking the interaction strength of the current user at the current moment as an element to construct an interaction strength sequence of the current user at the current moment";

[0054] If the second judgment result is yes, then the preset number of elements in the interaction strength sequence of the current user at the current moment are obtained as the strength stack of the current user, the current user is determined to be a pending interaction subject, the current user is updated, and the step of "when the current user is a new user, the interaction strength of the current user at the current moment is used as an element to construct the interaction strength sequence of the current user at the current moment" is returned to until all users participating in the multi-person collaborative design scenario at the current moment are traversed to obtain the strength stack of each pending interaction subject;

[0055] Based on the strength stacks of all pending interacting subjects, the interacting subject at the current moment is determined.

[0056] Optionally, determining the interaction subject at the current moment based on the strength stack of all pending interaction subjects includes:

[0057] Substitute the current time as variable t into the formula dec(t) = β t-1 The attenuation factor at the current moment is obtained from ; the variable t represents the moment, dec(t) is the attenuation factor at moment t, and β is a constant between 0 and 1;

[0058] Multiply each element in the strength stack of the current pending interaction subject by the attenuation factor at the current moment to obtain an updated strength stack;

[0059] Determine any pending interaction subject as the current pending interaction subject;

[0060] Determine that the mean of all elements in the update strength stack is the pending strength of the current pending interaction subject;

[0061] Determine whether the current pending interaction subject was an interaction subject at the last moment, and obtain a third determination result;

[0062] If the third judgment result is no, then the pending force is determined to be the update force of the current pending interactive subject;

[0063] If the third judgment result is yes, the product of the viscosity coefficient and the undetermined force is determined as the intermediate quantity;

[0064] Determine the sum of the pending force and the intermediate quantity as the updated force of the current pending interaction subject;

[0065] Update the current pending interaction subject, and return to the step of "determining that the average of all elements in the update strength stack is the pending strength of the current pending interaction subject" until all pending interaction subjects are traversed to obtain the updated strength of each pending interaction subject;

[0066] The pending interaction subject corresponding to the maximum update force is determined to be the interaction subject at the current moment.

[0067] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for switching interactive subjects of multi-person collaborative design.

[0068] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0069] The present application provides a method and device for switching interactive subjects in multi-person collaborative design. First, basic information of users is collected. In a multi-person collaborative design environment, the user's position and skeletal key point coordinates are collected based on a depth camera and users are numbered continuously. Secondly, a convolutional neural network is used to identify the user's posture level. Thirdly, the user's interaction strength is calculated. Finally, based on a multi-agent adversarial model, a natural switching of the current interactive subject is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 This is a flow chart of a method for switching subjects of multi-person collaborative design interaction in one embodiment of the present application.

[0072] Figure 2 This is a flow chart of a method for switching subjects of multi-person collaborative design interaction in another embodiment of the present application.

[0073] Figure 3 This is a technical framework diagram for collecting user features in a multi-person collaborative design environment in one embodiment of the present application.

[0074] Figure 4 This is a training flowchart of a user individual posture level recognition model in one embodiment of the present application.

[0075] Figure 5 This is a flowchart of individual posture level recognition of a certain user at the current moment in one embodiment of the present application.

[0076] Figure 6 This is a flow chart of a method for calculating interaction strength based on multi-feature fusion in one embodiment of the present application.

[0077] Figure 7 This is a flowchart of the natural switching of interactive subjects in one embodiment of the present application.

[0078] Figure 8 It is a structural schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0081] In an exemplary embodiment, a method for switching subjects of multi-person collaborative design interaction is provided, and the method for switching subjects of multi-person collaborative design interaction is applied to a human-computer interaction system supporting multi-person collaborative design scenarios, and the human-computer interaction system supporting multi-person collaborative design scenarios includes a depth camera. The depth camera is used to obtain a multi-person collaborative design scene image sequence at a preset frequency.

[0082] like Figure 1 As shown, the method for switching subjects of multi-person collaborative design interaction includes:

[0083] Step 101: Acquire a multi-person collaborative design scene image at the current moment as the current image.

[0084] Step 102: Determine all users who currently participate in the multi-person collaborative design scene based on the current image, as well as the posture level, displacement, and speed of each user at the current moment.

[0085] Step 103: Based on the posture level, displacement and speed of the same user at the current moment, determine the interaction intensity of the corresponding user at the current moment.

[0086] Step 104: Determine the interaction subject at the current moment based on the interaction strength of all users participating in the multi-person collaborative design scenario at the current moment.

[0087] Step 105: When the interaction subject at the current moment is different from the interaction subject at the previous moment, switch the interaction subject.

[0088] Step 102 includes:

[0089] Step 102-1: Use the Openpose tool to extract the first posture data of each user in the current image. The first posture data includes the coordinates of multiple key points of the same user in the camera coordinate system.

[0090] Step 102 - 2 : converting the first posture data of all users into the world coordinate system to obtain the second posture data corresponding to each user.

[0091] Step 102 - 3 : Determine any second body shape data as current body shape data.

[0092] Step 102 - 4 : Determine that the user corresponding to the current body posture data is the current user.

[0093] Step 102-5: Based on the current body posture data, determine the displacement and speed of the current user at the current moment.

[0094] Step 102-6: Determine the occlusion degree of the current user according to the interval of the number of key points in the current body data. The interval corresponds to the occlusion degree one by one. The occlusion degree includes an unrecognizable occlusion degree and multiple recognizable occlusion degrees.

[0095] Step 102-7: When the occlusion degree of the current user is an unrecognizable occlusion degree, update the current body posture data and return to step 102-4.

[0096] Step 102-8: When the occlusion degree of the current user is a recognizable occlusion degree, determine that the individual posture level recognition model corresponding to the occlusion degree of the current user is the current individual posture level recognition model. The recognizable occlusion degree corresponds to the individual posture level recognition model one-to-one. The individual posture level recognition model is obtained by training the convolutional neural network using a plurality of second body posture data corresponding to the recognizable occlusion degree.

[0097] Step 102-9: Input the current body posture data into the current individual posture grade recognition model to obtain the posture grade of the current user at the current moment.

[0098] Step 102 - 10 : Update the current posture data and return to step 102 - 4 until all the second posture data corresponding to the current image are traversed to obtain the posture level of each user at the current moment.

[0099] After step 102-4, the method further includes:

[0100] Step 102-4-1: Using a continuity detection algorithm, determine whether the current user exists in the multi-person collaborative design scene image at the previous moment, and obtain a first determination result.

[0101] Step 102-4-2: If the first judgment result is no, determine that the current user is a new user, and determine that the user number of the current user is the maximum user number at the previous moment plus 1.

[0102] Step 102-4-3: If the first judgment result is yes, the current user is determined to be an existing user, and the user ID of the current user at the previous moment is used.

[0103] Before step 101, the method further includes:

[0104] Step 106: construct multiple convolutional neural networks. The number of convolutional neural networks is equal to the number of recognizable occlusion levels. The convolutional neural network includes a convolutional layer, a maximum pooling layer, a flattening layer, a first fully connected layer, and a second fully connected layer connected in sequence.

[0105] Step 107: Determine any recognizable occlusion degree as the current recognizable occlusion degree.

[0106] Step 108: Acquire a plurality of second body posture data of currently identifiable occlusion degrees.

[0107] Step 109: marking posture levels corresponding to a plurality of second posture data of currently identifiable occlusion degrees.

[0108] Step 1010: Taking multiple second body posture data of the currently recognizable occlusion degree as input and taking the posture level as output, a convolutional neural network is trained to obtain an individual posture level recognition model corresponding to the currently recognizable occlusion degree.

[0109] Step 1011: Update the current recognizable occlusion degree, and return to step 108 until all recognizable occlusion degrees are traversed to obtain an individual posture level recognition model corresponding to each recognizable occlusion degree.

[0110] Step 102-5 includes:

[0111] Step 102-5-1: When the current user is a new user, update the current body data and return to step 102-4.

[0112] Step 102-5-2: When the current user is an existing user, the second posture data of the current user in the multi-person collaborative design scene image at a first time interval before the current moment is obtained as the historical second posture data.

[0113] Step 102-5-3: Determine the coordinate average of all key points in the historical second body posture data in the world coordinate system as the historical position of the current user.

[0114] Step 102-5-4: Determine the coordinate mean of all key points in the current body data in the world coordinate system as the current position of the current user history.

[0115] Step 102-5-5: Determine the displacement of the current user at the current moment according to the historical location and current location of the current user.

[0116] Step 102-5-6: Determine the ratio of the displacement of the current user at the current moment to the first time interval, which is the speed of the current user at the current moment.

[0117] The interaction strength is:

[0118]

[0119] Among them, L i，t G represents the interaction intensity of the i-th user participating in the multi-person collaborative design scenario at time t. i represents the posture level of the i-th user participating in the multi-person collaborative design scenario at time t. * represents a multiplication operation. e represents a natural constant, which is approximately equal to 2.71828. ΔP i V represents the displacement of the i-th user participating in the multi-person collaborative design scenario at time t. i Represents the speed of the i-th user participating in the multi-person collaborative design scenario at time t.

[0120] Step 104 includes:

[0121] Step 104 - 1 : Determine any user as the current user.

[0122] Step 104 - 2 : When the current user is a new user, the interaction intensity of the current user at the current moment is used as an element to construct an interaction intensity sequence of the current user at the current moment.

[0123] Step 104 - 3 : When the current user is an existing user, obtain the interaction intensity sequence of the current user at the last moment.

[0124] Step 104 - 4 : Add the interaction strength of the current user at the current moment as the last element to the interaction strength sequence of the current user at the previous moment, to obtain the interaction strength sequence of the current user at the current moment.

[0125] Step 104 - 5 : Determine whether the number of elements in the interaction intensity sequence of the current user at the current moment reaches a preset number, and obtain a second determination result.

[0126] Step 104-6: If the second judgment result is no, update the current user and return to step 104-2.

[0127] Step 104-7: If the second judgment result is yes, then obtain the preset number of elements in the interaction strength sequence of the current user at the current moment as the strength stack of the current user, determine the current user as the pending interaction subject, update the current user, and return to step 104-2 until all users participating in the multi-person collaborative design scenario at the current moment are traversed to obtain the strength stack of each pending interaction subject.

[0128] Step 104 - 8 : Determine the interacting subject at the current moment based on the strength stacks of all the interacting subjects to be determined.

[0129] Step 104-8 includes:

[0130] Step 104-8-1: Substitute the current time as variable t into the formula dec(t) = β t-1 The attenuation factor at the current moment is obtained from . The variable t represents the moment, dec(t) is the attenuation factor at moment t, and β is a constant between 0 and 1.

[0131] Step 104-8-2: Multiply each element in the strength stack of the current pending interaction subject by the attenuation factor at the current moment to obtain an updated strength stack.

[0132] Step 104-8-3: Determine any pending interaction subject as the current pending interaction subject.

[0133] Step 104-8-4: Determine the average value of all elements in the update strength stack as the pending strength of the current pending interaction subject.

[0134] Step 104-8-5: Determine whether the current pending interaction subject was an interaction subject at the last moment, and obtain a third determination result.

[0135] Step 104-8-6: If the third judgment result is no, then the pending force is determined to be the update force of the current pending interactive subject.

[0136] Step 104-8-7: If the third judgment result is yes, the product of the viscosity coefficient and the undetermined force is determined as the intermediate quantity.

[0137] Step 104-8-8: Determine the sum of the pending force and the intermediate quantity as the updated force of the current pending interactive subject.

[0138] Step 104-8-9: Update the current pending interaction subject, and return to step 104-8-4 until all pending interaction subjects are traversed to obtain the update force of each pending interaction subject.

[0139] Step 104-8-10: Determine that the pending interaction subject corresponding to the maximum update force is the interaction subject at the current moment.

[0140] In an exemplary embodiment, a method for switching interactive subjects in multi-person collaborative design is provided, and a method for representing the user interaction intensity based on characteristics such as user posture level, displacement, and speed is used to achieve quantitative representation of user interaction intensity. On this basis, a dynamic switching model of interactive agents in multi-person confrontation is constructed to achieve dynamic switching of interactive subjects. This embodiment is aimed at the design process of designers in a multi-person collaborative design environment. By collecting and analyzing the kinematic model and posture data of designers, the interaction intensity of designers is calculated, and finally the natural switching of interactive subjects is achieved. It has positive significance for the development of user state perception and interaction process in a multi-person collaborative design environment.

[0141] like Figure 2 , a method for switching interactive subjects in multi-person collaborative design, specifically comprising the following steps:

[0142] Step 1: Collection of basic user information.

[0143] The collected data includes the user's spatial position, the coordinates of the skeleton key points, and the users are numbered continuously. The technical framework used is as follows: Figure 3 As shown:

[0144] Hardware device: Intel Real Sense D455 depth camera.

[0145] Software environment: Openpose environment + Pycharm software integrated development environment.

[0146] 1) Use Openpose to extract the skeleton key points of the user in the scene.

[0147] Each frame of image captured by the depth camera is used as the data input of Openpose. According to the Openpose algorithm, the user will get the key point position information of the human joint extracted by the user, and the key point position and connection relationship can be annotated. In this process, the Openpose parameter setting sets the input camera resolution to 1280×720, and the input mode turns on the key point detection of the torso and hand at the same time. Each user can output 25 key point information. Adjust the model network output size according to the computer processing performance, such as 256×176. Finally, save the output results of Openpose as a list or JSON file.

[0148] 2) Use coordinate transformation to calculate the spatial three-dimensional coordinates of the user's skeleton key points.

[0149] The depth camera uses a left-handed coordinate system, with the origin of the coordinate system located at the optical center of the camera. The origin of the depth image coordinate system is located in the upper left corner of the image, the positive direction of the x-axis is to the right of the camera's field of view, the positive direction of the y-axis is to the bottom of the camera's field of view, and the positive direction of the z-axis is to the front of the camera's field of view. Obtaining the user's three-dimensional coordinates is divided into the following two steps:

[0150] ① Get the 3D coordinates of the user in the camera coordinate system: Combined with the Openpose algorithm introduced above, the 2D coordinates P of each posture key point of each user in the D455 field of view can be obtained. ori (x, y). The screen 2D coordinates of the user's No. 1 posture key point at this moment are used as the input of the D455 depth camera to obtain the position P of the user's No. 1 posture key point in the camera coordinate system. camera (x,y,z).

[0151] ②Convert the camera coordinates into world coordinates. The transformation formula used is as follows:

[0152]

[0153] Among them, (X W ,Y W ,Z W ) is the coordinate of the key point in the camera coordinate system, (X G ,Y G ,Z G ) is the coordinate of the key point in the world coordinate system, R and t are the 3×3 rotation matrix and 3×1 translation vector respectively, depending on the actual situation.

[0154] 3) User's continuity number.

[0155] Suppose a user's two-dimensional location data is (x i ,y i ), the basic idea of ​​the continuity detection algorithm is to determine whether the user's position is continuous by calculating the Euclidean distance between adjacent position data. Specifically, for adjacent position data (x i ,y i ) and (x i+1 ,y i+1 ), the Euclidean distance △d is calculated as shown in formula (2).

[0156]

[0157] If the △d between them is less than a preset threshold d th, then they are considered to be continuous, as shown in formula (3). If the location data of a user is continuous, then its number will not change; if a new user enters the screen, or the location data of a user is not continuous with the later location data, then a new number needs to be added.

[0158]

[0159] Step 2: Identify the user's individual posture level.

[0160] For user individual posture level recognition, the user's posture level should be defined first. The relevant definitions of this embodiment are shown in Table 1:

[0161] Table 1 User posture definition table

[0162]

[0163]

[0164]

[0165] Based on the above definition and the human key point data collected by Openpose, a dataset containing the coordinates of human key points and posture levels is constructed. In this process, the sample data is divided as shown in Table 2 according to the different occlusion levels of the user.

[0166] Table 2: Sample data partition table

[0167] Serial number The number of invalid points Type of sample ① Less than or equal to 4 Basic unobstructed data sample ② More than 4 and less than 10 A small number of occlusion data samples ③ More than 10 and less than 15 Most occlusion data samples ④ Greater than or equal to 15 Invalid sample

[0168] According to the different degrees of occlusion of users in the multi-person collaborative design environment, three convolutional neural networks are built and trained. The training process is as follows: Figure 4 As shown:

[0169] The Sequential model of Keras is used in the training process of the neural network. The model contains a convolution layer, a maximum pooling layer, a flattening layer, and two fully connected layers. In this model, the present invention first adds a convolution layer with 64 filters and 3 convolution kernels. ReLU is used as the activation function, and the input shape is (25, 2). Next is a maximum pooling layer with a pooling size of 2. Then there is a flattening layer, which flattens the three-dimensional array output by the convolution layer into a two-dimensional matrix. Then there are two fully connected layers, the first one has 100 neurons and uses the ReLU activation function, and the second one has 10 neurons and uses the Softmax activation function.

[0170] After training the three convolutional neural networks, the recognition process of the individual posture level of a user at the current moment is as follows: Figure 5As shown: first determine the occlusion type based on the user's skeleton key point data collected by Openpose; input the skeleton key point data into the corresponding type of user individual posture level recognition model; finally, output the user's posture level at the current moment.

[0171] Step 3: Calculate user interaction intensity.

[0172] The user's interaction intensity is affected by multiple factors. Specifically, the interaction intensity is affected by the user's posture level, user speed, and user displacement. Figure 6 Therefore, this step will be subdivided into the following two parts: 1) calculation of the user's kinematic model; 2) calculation of the interaction strength based on multi-feature fusion.

[0173] 1) Kinematic model of the user.

[0174] After obtaining the user's three-dimensional coordinates, the Euclidean distance can be used to calculate the user's displacement within a period of time, and the first-order derivative of the displacement is used as the user's velocity.

[0175] The displacement of a user over a period of time can be expressed by Euclidean distance:

[0176]

[0177] in, is the three-dimensional coordinate of the i-th user at time t2, is the three-dimensional coordinate of the user at time t1.

[0178] The average speed of the user during this period can be expressed as:

[0179]

[0180] 2) Strength calculation of multi-feature fusion.

[0181] In order to better calculate the user's interaction characteristics, this embodiment introduces a nonlinear exponential function to combine various characteristics (user posture level, speed and displacement). After experimental verification and theoretical analysis, the calculation formula (6) of the interaction intensity is as follows:

[0182]

[0183] Step 4: Natural switching of interaction subjects.

[0184] In order to optimize the interactive subject confrontation switching, this embodiment proposes a dynamic interactive subject switching method based on multi-subject confrontation. This method takes into account the power of each user's competition for control, the stickiness factor, and the attenuation factor to achieve smooth and adaptive subject switching. The specific implementation process is as follows: Figure 7 As shown:

[0185] First, an intensity stack is established for each user. At each time step t, this application records the interaction intensity I of each user i. i (t). The intensity stack of user i is represented as I i = {I i (1), I i (2), ..., I i (t)}. Next, the average interaction strength of each user is calculated to estimate its power F in competing for control:

[0186]

[0187] Among them, |I i | represents the length of the strength stack of user i.

[0188] Secondly, the viscosity coefficient α is introduced to make it more difficult for the current user to lose control at the next moment. This application can combine the viscosity coefficient with the user's force F to calculate an adjusted force F':

[0189] F′(t)=F i (t)·(1+α·δ(i,C t )) (8)

[0190] Among them, C t represents the controlling user at the current time t, δ(i, C t ) is the indicator function, when i=C t 1 if the value is 0, otherwise it is 0.

[0191] Then, the attenuation factor dec is introduced to gradually reduce the influence of the early interaction strength on the current strength. Let β be a constant between 0 and 1. This application can define the attenuation factor as:

[0192] dec(t)=β t-1 (9)

[0193] Now, this embodiment can update the user's intensity stack according to the decay factor:

[0194] I′ i (t) = dec(t)·I i (t) (10)

[0195] Next, this embodiment needs to recalculate the adjusted force F'. This can be done by adding the updated strength stack I' i This can be achieved by substituting (t) into the calculation formula of F'.

[0196] Finally, at each time step t, the application assigns steering rights to all users according to the adjusted force F':

[0197] Agent t = argmax(F′ i (t)), i=1, 2,..., N (11)

[0198] The method provided in this implementation completes the dynamic switching of interactive subjects based on multi-subject confrontation, comprehensively considering the power of users competing for control rights, viscosity coefficient and attenuation factor, and can achieve more fair and stable subject switching. By adjusting the parameters α and β, the present application can implement different switching strategies to meet the needs of different scenarios. Virtual design technology in a multi-person collaborative design environment can not only help people perform tedious tasks such as graphics processing and strength calculation, but also help designers to carry out innovative designs. Such design methods are of great significance to shortening the product development cycle. In addition, it can free designers from the keyboard and mouse, making it easier to interact with the design object in a more natural and intuitive way. This embodiment solves the two major technical problems of how to calculate the interaction intensity and how to naturally switch the interactive subjects in a multi-person collaborative design environment. This goal is achieved mainly by using a multi-feature fusion method and multi-subject confrontation.

[0199] In an exemplary embodiment, a multi-person collaborative design interactive subject switching device is provided, which mainly includes the following four modules:

[0200] Information collection module: Starting from the depth vision camera to collect information in the room, through the Openpose algorithm, output the user number and the three-dimensional coordinates of the skeleton key points.

[0201] Individual posture level recognition module: Set the user's individual posture level from low to high, and make the collected data into a data set. Build and train three convolutional neural networks according to the different degrees of user occlusion to achieve real-time recognition of the user's individual posture level.

[0202] Interaction intensity calculation module: A nonlinear interaction intensity calculation model based on the user's displacement, speed and individual posture level is built.

[0203] Automatic switching module for interactive subjects: A dynamic switching method for interactive subjects based on multi-subject confrontation is built, and viscosity factors and attenuation factors are introduced.

[0204] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for switching interactive subjects of multi-person collaborative design is implemented.

[0205] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0206] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0207] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0209] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0210] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0211] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0212] The present application uses specific examples to illustrate the principles and implementation methods of the present application. The above examples are only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for switching subjects of multi-person collaborative design interaction, characterized in that: The multi-person collaborative design interaction subject switching method is applied to a human-computer interaction system supporting a multi-person collaborative design scenario, wherein the human-computer interaction system supporting a multi-person collaborative design scenario includes a depth camera; The depth camera is used to obtain a multi-person collaborative design scene image sequence at a preset frequency; The multi-person collaborative design interaction subject switching method comprises: Acquire the multi-person collaborative design scene image at the current moment as the current image; Determine all users participating in the multi-person collaborative design scene at the current moment based on the current image, as well as the posture level, displacement and speed of each user at the current moment; Based on the posture level, displacement and speed of the same user at the current moment, determine the interaction intensity of the corresponding user at the current moment; Based on the interaction intensity of all users participating in the multi-person collaborative design scenario at the current moment, determine the interaction subject at the current moment; When the interaction subject at the current moment is different from the interaction subject at the previous moment, the interaction subject is switched.

2. The method for switching subjects of multi-person collaborative design interaction according to claim 1, characterized in that: Determine all users participating in the multi-person collaborative design scene at the current moment based on the current image, as well as the posture level, displacement and speed of each user at the current moment, including: Extracting first posture data of each user in the current image using the Openpose tool; the first posture data includes coordinates of multiple key points of the same user in a camera coordinate system; The first posture data of all users are converted into the world coordinate system to obtain the second posture data corresponding to each user; Determine any second body shape data as current body shape data; Determine that the user corresponding to the current body posture data is the current user; Based on the current body posture data, determining the displacement and speed of the current user at the current moment; Determine the occlusion degree of the current user according to the interval where the number of key points in the current body posture data is located; the interval corresponds to the occlusion degree one by one; the occlusion degree includes an unrecognizable occlusion degree and multiple recognizable occlusion degrees; When the occlusion degree of the current user is an unrecognizable occlusion degree, the current body posture data is updated and the process returns to the step of "determining that the user corresponding to the current body posture data is the current user"; When the occlusion degree of the current user is a recognizable occlusion degree, determining that the individual posture level recognition model corresponding to the occlusion degree of the current user is the current individual posture level recognition model; the recognizable occlusion degree corresponds to the individual posture level recognition model one-to-one; the individual posture level recognition model is obtained by training a convolutional neural network using a plurality of second body posture data corresponding to the recognizable occlusion degree; Inputting the current body posture data into a current individual posture grade recognition model to obtain the posture grade of the current user at the current moment; Update the current posture data and return to step "determine that the user corresponding to the current posture data is the current user" until all the second posture data corresponding to the current image are traversed to obtain the posture level of each user at the current moment.

3. The method for switching subjects of multi-person collaborative design interaction according to claim 2, characterized in that: After determining that the user corresponding to the current body posture data is the current user, the method further includes: Using a continuity detection algorithm, it is determined whether the current user exists in the multi-person collaborative design scene image at the previous moment, and a first determination result is obtained; If the first judgment result is no, the current user is determined to be a new user, and the user number of the current user is determined to be the maximum user number at the previous moment plus 1; If the first judgment result is yes, it is determined that the current user is an existing user, and the user number of the current user at the previous moment is used.

4. The method for switching subjects of multi-person collaborative design interaction according to claim 2, characterized in that: Before obtaining the multi-person collaborative design scene image at the current moment as the current image, the following is also included: Constructing a plurality of convolutional neural networks; the number of the convolutional neural networks is equal to the number of the recognizable occlusion degrees; Determine any recognizable occlusion degree as the current recognizable occlusion degree; Acquire a plurality of second body posture data of currently recognizable occlusion degree; Annotating posture levels corresponding to a plurality of second posture data of currently identifiable occlusion degrees; Taking multiple second body posture data of the currently recognizable occlusion degree as input and taking the posture level as output, the convolutional neural network is trained to obtain an individual posture level recognition model corresponding to the currently recognizable occlusion degree; Update the current recognizable occlusion degree, and return to step "obtaining multiple second posture data of the current recognizable occlusion degree" until all recognizable occlusion degrees are traversed to obtain the individual posture level recognition model corresponding to each recognizable occlusion degree.

5. The method for switching subjects of multi-person collaborative design interaction according to claim 2, characterized in that: The convolutional neural network includes a convolutional layer, a maximum pooling layer, a flattening layer, a first fully connected layer and a second fully connected layer which are connected in sequence.

6. The method for switching subjects of multi-person collaborative design interaction according to claim 3, characterized in that: Determining the displacement and speed of the current user at the current moment based on the current body posture data includes: When the current user is a new user, update the current body shape data and return to the step of "determining that the user corresponding to the current body shape data is the current user"; When the current user is an existing user, obtaining the second posture data of the current user in the multi-person collaborative design scene image at a first time interval before the current moment as the historical second posture data; Determine the coordinate mean of all key points in the historical second body data in the world coordinate system as the historical position of the current user; Determine the coordinate mean of all key points in the current body data in the world coordinate system as the current position of the current user history; Determine the displacement of the current user at the current moment based on the historical location and current location of the current user; The ratio of the displacement of the current user at the current moment to the first time interval is determined as the speed of the current user at the current moment.

7. The method for switching subjects of multi-person collaborative design interaction according to claim 1, characterized in that: The interaction strength is: Among them, L i，t represents the interaction intensity of the i-th user participating in the multi-person collaborative design scenario at time t; G i represents the posture level of the i-th user participating in the multi-person collaborative design scenario at time t; * represents multiplication operation; e represents a natural constant; ΔP i represents the displacement of the i-th user participating in the multi-person collaborative design scenario at time t; V i Represents the speed of the i-th user participating in the multi-person collaborative design scenario at time t.

8. The method for switching subjects of multi-person collaborative design interaction according to claim 3, characterized in that: Based on the interaction intensity of all users participating in the multi-person collaborative design scenario at the current moment, the interaction subject at the current moment is determined, including: Determine any user as the current user; When the current user is a new user, the interaction intensity of the current user at the current moment is used as an element to construct the interaction intensity sequence of the current user at the current moment; When the current user is an existing user, obtain the interaction intensity sequence of the current user at the previous moment; Add the interaction strength of the current user at the current moment as the last element to the interaction strength sequence of the current user at the previous moment to obtain the interaction strength sequence of the current user at the current moment; Determine whether the number of elements in the interaction intensity sequence of the current user at the current moment reaches a preset number, and obtain a second determination result; If the second judgment result is no, the current user is updated, and the process returns to step "when the current user is a new user, taking the interaction intensity of the current user at the current moment as an element to construct an interaction intensity sequence of the current user at the current moment"; If the second judgment result is yes, then the preset number of elements in the interaction strength sequence of the current user at the current moment are obtained as the strength stack of the current user, the current user is determined to be a pending interaction subject, the current user is updated, and the process returns to step "when the current user is a new user, the interaction strength of the current user at the current moment is used as an element to construct the interaction strength sequence of the current user at the current moment" until all users participating in the multi-person collaborative design scenario at the current moment are traversed to obtain the strength stack of each pending interaction subject; Based on the strength stacks of all pending interacting subjects, the interacting subject at the current moment is determined.

9. The method for switching subjects of multi-person collaborative design interaction according to claim 8, characterized in that: Based on the strength stack of all pending interacting subjects, the interacting subject at the current moment is determined, including: Substitute the current time as variable t into the formula dec(t) = β t-1 The attenuation factor at the current moment is obtained from ; the variable t represents the moment, dec(t) is the attenuation factor at moment t, and β is a constant between 0 and 1; Multiply each element in the strength stack of the current pending interaction subject by the attenuation factor at the current moment to obtain an updated strength stack; Determine any pending interaction subject as the current pending interaction subject; Determine that the mean of all elements in the update strength stack is the pending strength of the current pending interaction subject; Determine whether the current pending interaction subject was an interaction subject at the last moment, and obtain a third determination result; If the third judgment result is no, then the pending force is determined to be the update force of the current pending interactive subject; If the third judgment result is yes, the product of the viscosity coefficient and the undetermined force is determined as the intermediate quantity; Determine the sum of the pending force and the intermediate quantity as the updated force of the current pending interaction subject; Update the current pending interaction subject, and return to step "determine that the average of all elements in the update strength stack is the pending strength of the current pending interaction subject" until all pending interaction subjects are traversed to obtain the updated strength of each pending interaction subject; The pending interaction subject corresponding to the maximum update force is determined to be the interaction subject at the current moment.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for switching interactive subjects of multi-person collaborative design according to any one of claims 1 to 9.

Images