Model construction method, prediction method and device based on multi-target object interaction

Abstract

The invention relates to the technical field of visual perception processing, and discloses a model construction method, prediction method and device based on multi-target object interaction, and the method comprises the steps: representing the interaction relation among a plurality of target objects through the first distance information, the second distance information, the first posture information and the second posture information of the plurality of target objects; the multiple target objects are automatically associated in the same stereoscopic vision interaction model, the stereoscopic vision interaction model corresponding to the single target object does not need to be repeatedly reconstructed for multiple times, and the interaction relation does not need to be inferred continuously, so that the tedious process of model construction is simplified; and the accuracy of model construction is improved.

Description

Technical Field

[0001] The present invention relates to the field of visual perception processing technology, and in particular to a model building method, a prediction method and a device based on multi-target object interaction. Background Art

[0002] The stereoscopic visual interaction model uses images as input to build a stereoscopic model between the visual target object (such as the human body) and the target object that interacts with the surrounding (such as objects). This technology has important application value in the fields of digital twins and digital humans.

[0003] The construction of stereoscopic visual interaction model requires the realization of stereoscopic reconstruction of different target objects in a rich environment and complex visual information, so this task is very challenging. Especially for flexible target objects, due to their different shapes and the diversity of interaction forms with surrounding target objects, it is currently impossible to reconstruct a relatively complete stereoscopic visual interaction model.

[0004] Related technologies use a single target object to build a stereoscopic visual interaction model repeatedly, and after completing multiple model constructions, infer the interaction relationship between different target objects based on a priori methods. Since each stereoscopic visual interaction model obtained by the reconstruction process of a single target object is isolated, it is not conducive to the use of semantic information between different stereoscopic visual interaction models, and the priori method relies on the interaction reasoning results between different target objects and relies on predefined weights, resulting in inaccurate construction of the stereoscopic visual interaction model. Summary of the invention

[0005] In view of this, the present invention provides a model construction method, a prediction method and a device based on multi-target object interaction to solve the problem of inaccurate construction of stereoscopic vision interaction models in related technologies.

[0006] In a first aspect, the present invention provides a model construction method based on multi-target object interaction, the method comprising:

[0007] Based on the original video image, a first image feature corresponding to a first target object and a second image feature corresponding to a second target object are acquired; wherein the first target object is a flexible target object with life, and the second target object is a rigid target object without life;

[0008] Acquire a first position of the first target object in the stereoscopic visual space, a second position of the second target object in the stereoscopic visual space, and an interaction position of the first target object and the second target object;

[0009] Based on the first image feature and the second image feature, obtaining first distance information between the first position and the interaction position, second distance information between the second position and the interaction position, first posture information of the active organ of the first target object and second posture information of the second target object;

[0010] Based on the first distance information, the second distance information, the first posture information, and the second posture information, a stereoscopic visual interaction model of the first target object interacting with the second target object is constructed.

[0011] The disclosed embodiment utilizes the first distance information, the second distance information, the first posture information, and the second posture information of the multiple target objects to characterize the interaction relationship between the multiple target objects, and then constructs a stereoscopic visual interaction model. Since the multiple target objects are automatically associated in the same stereoscopic visual interaction model, there is no need to repeatedly rebuild the stereoscopic visual interaction model corresponding to a single target object and continuously infer the interaction relationship. Therefore, the disclosed embodiment not only simplifies the tedious process of model construction, but also improves the accuracy of model construction.

[0012] In an optional implementation, obtaining a first image feature corresponding to the first target object and a second image feature corresponding to the second target object based on the original video image includes:

[0013] Collecting original video images;

[0014] Processing the original video image to obtain a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object;

[0015] A first image feature is extracted from the two-dimensional mask corresponding to the first target object through a convolutional coding network, and a second image feature is extracted from the two-dimensional mask corresponding to the first target object.

[0016] The disclosed embodiment utilizes a convolutional coding network to rapidly implement decoding processing of original video images.

[0017] In an optional implementation, based on the first image feature and the second image feature, obtaining first distance information between the first target object and the target interaction point, second distance information between the second target object and the target interaction point, first posture information of the active organ of the first target object, and second posture information of the second target object includes:

[0018] Based on the first image feature and the second image feature, obtaining first distance information between the first target object and the target interaction point through a first decoding network;

[0019] Based on the first image feature and the second image feature, obtaining second distance information between the second target object and the target interaction point through a second decoding network;

[0020] Based on the first image feature and the second image feature, obtaining first posture information of the active organ of the first target object through a third decoding network;

[0021] Based on the first image feature and the second image feature, second posture information of the second target object is acquired through a fourth decoding network.

[0022] The disclosed embodiment processes the first image feature and the second image feature through four decoding networks in order to accurately extract four different feature information, thereby accurately capturing the associated features between the first target object and the second target object.

[0023] In some optional implementations, constructing a stereoscopic visual interaction model of the first target object interacting with the second target object based on the first distance information, the second distance information, the first posture information, and the second posture information includes:

[0024] Inputting the first distance information, the second distance information, the first posture information, and the second posture information into an initial stereoscopic vision interaction model for training;

[0025] combining the first distance information, the second distance information, the first posture information, and the second posture information using an energy function;

[0026] The stereoscopic vision interaction model is iteratively optimized using the optimization parameters until the optimization parameters reach a minimum value, and the iterative process ends.

[0027] The disclosed embodiment continuously optimizes the stereoscopic vision interaction model based on the first distance information, the second distance information, the first posture information and the second posture information, thereby improving the construction accuracy of the stereoscopic vision interaction model. In addition, using the trained stereoscopic vision interaction model, it is possible to predict whether there is an interaction relationship between any two objects to be tested.

[0028] According to a second aspect, an embodiment of the present disclosure provides an interactive relationship prediction method based on multi-target object interaction, the method comprising:

[0029] Based on the video image to be tested, a third image feature corresponding to the first object to be tested and a fourth image feature corresponding to the second object to be tested are obtained; wherein the first object to be tested is a flexible target object with life, and the second object to be tested is a rigid target object without life;

[0030] Acquire a third position of the first object to be measured in the stereoscopic visual space, a fourth position of the second object to be measured in the stereoscopic visual space, and an interaction position of the first object to be measured and the second object to be measured;

[0031] Based on the third image feature and the fourth image feature, obtaining third distance information between the third position and the interaction position, fourth distance information between the fourth position and the interaction position, third posture information of the active organ of the first object to be measured and fourth posture information of the second object to be measured;

[0032] The third distance information, the fourth distance information, the third posture information, and the fourth posture information are input into the trained stereoscopic vision interaction model to predict whether there is an interaction relationship between the first object to be measured and the second object to be measured.

[0033] The disclosed embodiment utilizes a trained stereoscopic vision interaction model to accurately predict whether there is an interaction relationship between any two objects to be tested.

[0034] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the model building method for multi-target object interaction of the first aspect or any corresponding embodiment thereof, or the interaction relationship prediction method based on multi-target object interaction of the second aspect or any corresponding embodiment thereof.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for building a model of multi-target object interaction of the first aspect or any corresponding embodiment thereof, or the method for predicting interaction relationships based on multi-target object interaction of the second aspect.

[0036] In a fifth aspect, the present invention provides a computer program product comprising computer instructions, the computer instructions being used to enable a computer to execute the method for building a model of multi-target object interaction of the first aspect or any corresponding embodiment thereof, or the method for predicting interaction relationships based on multi-target object interaction of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of a model building method based on multi-target object interaction according to an embodiment of the present invention;

[0039] Figure 2is a flow chart of another model building method based on multi-target object interaction according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of decoding four different characteristic information according to an embodiment of the present invention;

[0041] Figure 4 is a flow chart of another model building method based on multi-target object interaction according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of an overall network architecture according to an embodiment of the present invention;

[0043] Figure 6 It is a simplified flow chart of a method for constructing a stereoscopic visual interaction model for multi-target object interaction according to an embodiment of the present invention;

[0044] Figure 7 is a flow chart of yet another model building method based on multi-target object interaction according to an embodiment of the present invention;

[0045] Figure 8 is a structural block diagram of a model building device based on multi-target object interaction according to an embodiment of the present invention;

[0046] Fig. 9 is a structural block diagram of an interactive relationship prediction device based on multi-target object interaction according to an embodiment of the present invention;

[0047] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] According to an embodiment of the present invention, an embodiment of a model building method based on multi-target object interaction is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] In this embodiment, a model construction method based on multi-target object interaction is provided, which can be used in mobile terminals such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a model building method based on multi-target object interaction according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0051] Step S101, based on the original video image, obtain a first image feature corresponding to a first target object and a second image feature corresponding to a second target object; wherein the first target object is a flexible target object with life, and the second target object is a rigid target object without life.

[0052] Specifically, the original video image is the original input image. The first target object is a flexible target object with life, for example, a human body or an animal body. Since the corresponding muscles or limbs of the human body or the animal body will change when they move, the first target object is a flexible target object; and the second target object is a rigid object without life, for example, a table or a chair or a blackboard or a wall or a bus. Since the table or the chair or the blackboard or the wall or the bus is an inanimate object and will not produce limb or muscle changes, the second target object is a rigid target object. Among them, the appearance representation of the first image feature and the second image feature in the stereoscopic visual space, for example, the human body bending feature and the table rotation feature.

[0053] Since a single target object is currently used to construct a stereoscopic visual interaction model, there is a problem of inaccurate model construction. Therefore, the disclosed embodiment obtains image feature information of multiple target objects from the original input image, and then obtains the mutual connection between the image features of the multiple target objects, thereby enhancing the accuracy of the stereoscopic visual interaction model construction.

[0054] Step S102 , obtaining a first position of the first target object in the stereoscopic visual space, a second position of the second target object in the stereoscopic visual space, and an interaction position between the first target object and the second target object.

[0055] Specifically, for example, the first target object is a human body, the second target object is a chair, the stereoscopic visual space is a room, the human body is sitting on a chair in the room reading a book, the first position is the position information of the human body in the room is A, the second position is the position information of the chair in the room is B, and the interaction position is the interaction position between the human body's buttocks and the chair, let the interaction position be C.

[0056] Step S103, based on the first image feature and the second image feature, obtain first distance information between the first position and the interaction position, second distance information between the second position and the interaction position, first posture information of the active organ of the first target object, and second posture information of the second target object.

[0057] Specifically, in the above example, the first position is A, the second position is B, the interactive position is C, the first distance information is the distance between A and C, and the second distance information is the distance between B and C. If the first target object is a human body, the movable organs of the first target object may be arms, legs, knees, soles of feet, palms, head, etc., and the first posture information may be a leg bending feature, a head tilting feature, an arm raising feature, etc.; if the second target object is a rigid object, including a table, a chair, or a wall, the second posture information may be the current appearance feature of the second target object.

[0058] Step S104: constructing a stereoscopic visual interaction model of the first target object interacting with the second target object based on the first distance information, the second distance information, the first posture information, and the second posture information.

[0059] Specifically, the first distance information, the second distance information, the first posture information, and the second posture information represent the significant features of the first target object and the second target object, as well as the interaction relationship between the two. These four feature information can be input into the stereoscopic vision interaction model for joint optimization training to obtain a trained stereoscopic vision interaction model.

[0060] The disclosed embodiment utilizes the first distance information, the second distance information, the first posture information, and the second posture information of the multiple target objects to characterize the interaction relationship between the multiple target objects, and then constructs a stereoscopic visual interaction model. Since the multiple target objects are automatically associated in the same stereoscopic visual interaction model, there is no need to repeatedly rebuild the stereoscopic visual interaction model corresponding to a single target object and continuously infer the interaction relationship. Therefore, the disclosed embodiment not only simplifies the tedious process of model construction, but also improves the accuracy of model construction.

[0061] In this embodiment, a model construction method based on multi-target object interaction is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 2 is a flow chart of a method for constructing a stereoscopic visual interaction model for multi-target object interaction according to an embodiment of the present invention. Figure 2 As shown, in the above step S101, based on the original video image, a first image feature corresponding to the first target object and a second image feature corresponding to the second target object are obtained. The process includes the following steps:

[0062] Step S201, capturing original video images.

[0063] Specifically, the original video image here is the original input image.

[0064] In a specific example, collecting an original video image includes:

[0065] Step a1, setting a preset sampling coefficient.

[0066] Step a2, sparsely sampling the video sequence based on a preset sampling coefficient to obtain RGB image data of a preset number of video frames, where the preset number of video frames is equal to the preset sampling coefficient.

[0067] Step a3: Use the RGB image data of a preset number of video frames as the original video image.

[0068] Specifically, the input video sequence is L frames long, which is a 3-channel RGB color image sequence, so the data dimension is 3×L×W×H. Sampling from the video sequence obtains the original input image with a dimension of 3×W×H.

[0069] Step S202: Process the original video image to obtain a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object.

[0070] Specifically, for example, grayscale processing is performed on the original video image to obtain a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target image. For example, the two-dimensional masks of the first target object and the second target object are 1×W×H respectively.

[0071] Step S203: extracting a first image feature from the two-dimensional mask corresponding to the first target object and extracting a second image feature from the two-dimensional mask corresponding to the first target object through a convolutional coding network.

[0072] Specifically, the convolutional coding network is a common network in machine learning.

[0073] In a specific example, the convolutional encoding network includes a convolutional layer, a batch normalization layer, and a ReLU layer. The input of the convolutional decoding network is a two-dimensional mask corresponding to a first target object and a two-dimensional mask corresponding to a second target object. The output is a first image feature and a second image feature. The first image feature and the second image feature serve as appearance representations of the original video image in a stereoscopic visual space.

[0074] Specifically, for example, when the first target object is a human body and the second target object is an object, the input of the convolutional coding network is a two-dimensional mask of the human body and object in the original input image, and the output is a frame-level convolutional coding feature, which represents the output semantic representation of the convolutional coding network.

[0075] The disclosed embodiment utilizes a convolutional coding network to rapidly implement decoding processing of original video images.

[0076] In some optional implementations, based on the first image feature and the second image feature, first distance information between the first target object and the target interaction point, second distance information between the second target object and the target interaction point, first posture information of the active organ of the first target object, and second posture information of the second target object are obtained. This includes:

[0077] Step b1, based on the first image feature and the second image feature, obtaining first distance information between the first target object and the target interaction point through a first decoding network.

[0078] Step b2: Based on the first image feature and the second image feature, obtaining second distance information between the second target object and the target interaction point through a second decoding network.

[0079] Step b3: Based on the first image feature and the second image feature, first posture information of the active organ of the first target object is obtained through a third decoding network.

[0080] Step b4: Based on the first image feature and the second image feature, obtaining second posture information of the second target object through a fourth decoding network.

[0081] like Figure 3 As shown, the first decoding network, the second decoding network, the third decoding network and the fourth decoding network are neural networks with the same function, which can be convolutional decoding networks, corresponding to the convolutional coding network, and mainly perform decoding processing actions of image features. The first decoding network decodes the first distance information from the first image feature and the second image feature, the second decoding network decodes the second distance information from the first image feature and the second image feature, the third decoding network decodes the first posture information from the first image feature and the second image feature, and the fourth decoding network decodes the second posture information from the first image feature and the second image feature. Use four decoding networks to process image coding features and obtain neural field information aligned with four pixels.

[0082] The disclosed embodiment processes the first image feature and the second image feature through four decoding networks in order to accurately extract four different feature information, thereby accurately capturing the associated features between the first target object and the second target object.

[0083] In this embodiment, a model construction method based on multi-target object interaction is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 4 is a flow chart of a method for constructing a model of multi-target object interaction according to an embodiment of the present invention. Figure 4 As shown, in the above step S104, based on the first distance information, the second distance information, the first posture information and the second posture information, a stereoscopic visual interaction model of the first target object interacting with the second target object is constructed. The process includes the following steps:

[0084] Step S401: input the first distance information, the second distance information, the first posture information, and the second posture information into an initial stereoscopic vision interaction model for training.

[0085] Specifically, the first distance information, the second distance information, the first posture information and the second posture information are used as input information in the stereoscopic vision interaction model.

[0086] Step S402: using an energy function to combine the first distance information, the second distance information, the first posture information and the second posture information.

[0087] Specifically, the energy function is a commonly used function in machine learning. During the training process of the stereoscopic vision interaction model, it is necessary to continuously adjust the model weights and bias parameters to minimize the value of the loss function. This is achieved by minimizing the value of the energy function. The smaller the value of the energy function, the more stable the neuron state.

[0088] Step S403, iteratively optimizing the stereoscopic vision interaction model using the optimization parameters until the optimization parameters reach a minimum value, and the iterative process ends.

[0089] Specifically, when the optimization parameter is the smallest, it means that the energy function value is the smallest, the optimization purpose is achieved, and then the stereoscopic visual interaction model is trained.

[0090] The disclosed embodiment continuously optimizes the stereoscopic visual interaction model based on the first distance information, the second distance information, the first posture information, and the second posture information, thereby improving the accuracy of constructing the stereoscopic visual interaction model. In addition, the trained stereoscopic visual interaction model can be used to predict whether there is an interaction relationship between any two objects to be tested. When the prediction result of the interaction relationship output by the stereoscopic visual interaction model is less than or equal to a preset threshold, it indicates that there is an interaction relationship between the two objects to be tested, otherwise, there is no interaction relationship between the two. For the specific prediction process, see the following content.

[0091] like Figure 5 As shown in FIG. 1 , a method for constructing a stereoscopic visual interaction model for multi-target object interaction in an embodiment of the present disclosure is shown, and a schematic diagram of the overall network architecture. Figure 6 The figure is a simplified flowchart of a method for constructing a stereoscopic visual interaction model for multi-target object interaction in an embodiment of the present disclosure.

[0092] In this embodiment, a method for predicting interaction relationships based on multi-target object interactions is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 7 is a flow chart of an interactive relationship prediction method according to an embodiment of the present invention. Figure 7 As shown, the process includes the following steps:

[0093] Step S701, based on the video image to be tested, obtain a third image feature corresponding to the first object to be tested and a fourth image feature corresponding to the second object to be tested; wherein the first object to be tested is a flexible target with life, and the second object to be tested is a rigid target without life.

[0094] Specifically, the video image to be tested is a video image to be planned for prediction, the first object to be tested and the second object to be tested are any two objects to be tested, and the specific method for obtaining the third image feature and the fourth image feature is the same as the specific method for obtaining the first image feature and the second image feature, see the above steps S201-S203, which will not be repeated here.

[0095] Step S702, obtaining a third position of the first object to be measured in the stereoscopic visual space, a fourth position of the second object to be measured in the stereoscopic visual space, and an interaction position between the first object to be measured and the second object to be measured.

[0096] Step S703, based on the third image feature and the fourth image feature, obtain the third distance information between the third position and the interaction position, the fourth distance information between the fourth position and the interaction position, the third posture information of the active organ of the first object to be measured, and the fourth posture information of the second object to be measured.

[0097] Specifically, the method for acquiring the third distance information, the fourth distance information, the third posture information and the fourth posture information refers to the above steps b1 to b4, which will not be repeated here.

[0098] Step S704: input the third distance information, the fourth distance information, the third posture information, and the fourth posture information into the trained stereoscopic vision interaction model to predict whether there is an interaction relationship between the first object to be measured and the second object to be measured.

[0099] Specifically, using the trained stereoscopic vision interaction model, it is possible to predict whether there is an interaction relationship between any two objects to be tested. When the prediction result of the interaction relationship output by the stereoscopic vision interaction model is less than or equal to a preset threshold, it indicates that there is an interaction relationship between the two objects to be tested; otherwise, there is no interaction relationship between the two objects.

[0100] In the present embodiment, a model building device for multi-target object interaction is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0101] This embodiment provides a model building device for multi-target object interaction, such as Figure 8 As shown, including:

[0102] The image feature acquisition module 801 is used to acquire a first image feature corresponding to a first target object and a second image feature corresponding to a second target object based on an original video image; wherein the first target object is a flexible target object with life, and the second target object is a rigid target object without life;

[0103] A spatial position acquisition module 802, used to acquire a first position of a first target object in a stereoscopic visual space, a second position of a second target object in a stereoscopic visual space, and an interaction position between the first target object and the second target object;

[0104] An interaction feature acquisition module 803 is used to acquire first distance information between the first position and the interaction position, second distance information between the second position and the interaction position, first posture information of the active organ of the first target object, and second posture information of the second target object based on the first image feature and the second image feature;

[0105] The interaction model building module 804 is used to build a stereoscopic visual interaction model of the interaction between the first target object and the second target object based on the first distance information, the second distance information, the first posture information and the second posture information.

[0106] This embodiment provides an interactive relationship prediction device based on multi-target object interaction. Fig. 9 As shown, including:

[0107] The feature acquisition module 901 is used to acquire a third image feature corresponding to the first object to be tested and a fourth image feature corresponding to the second object to be tested based on the video image to be tested; wherein the first object to be tested is a flexible target object with life, and the second object to be tested is a rigid target object without life;

[0108] A spatial position acquisition module 902 is used to acquire a third position of the first object to be measured in the stereoscopic visual space, a fourth position of the second object to be measured in the stereoscopic visual space, and an interaction position of the first object to be measured and the second object to be measured;

[0109] An interaction position acquisition module 903 is used to acquire, based on the third image feature and the fourth image feature, third distance information between the third position and the interaction position, fourth distance information between the fourth position and the interaction position, third posture information of the active organ of the first object to be measured, and fourth posture information of the second object to be measured;

[0110] The interaction relationship prediction module 904 is used to input the third distance information, the fourth distance information, the third posture information, and the fourth posture information into the trained stereoscopic vision interaction model to predict whether there is an interaction relationship between the first object to be measured and the second object to be measured.

[0111] In some optional implementations, the feature acquisition module to be tested includes:

[0112] An image acquisition submodule, used for acquiring original video images;

[0113] An image processing submodule, used to process the original video image to obtain a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object;

[0114] The feature extraction submodule is used to extract the first image feature from the two-dimensional mask corresponding to the first target object through a convolutional coding network, and to extract the second image feature from the two-dimensional mask corresponding to the first target object.

[0115] In some optional implementations, the interaction feature acquisition module 803 includes:

[0116] A first acquisition submodule, configured to acquire first distance information between a first target object and a target interaction point through a first decoding network based on the first image feature and the second image feature;

[0117] A second acquisition submodule, configured to acquire second distance information between a second target object and a target interaction point through a second decoding network based on the first image feature and the second image feature;

[0118] A third acquisition submodule is used to acquire first posture information of the active organ of the first target object through a third decoding network based on the first image feature and the second image feature;

[0119] The fourth acquisition submodule is used to acquire second posture information of the second target object through a fourth decoding network based on the first image feature and the second image feature.

[0120] In some optional implementations, the interaction model building module 804 includes:

[0121] A training submodule, used for inputting the first distance information, the second distance information, the first posture information and the second posture information into an initial stereoscopic vision interaction model for training;

[0122] A combining submodule, for combining the first distance information, the second distance information, the first posture information, and the second posture information by using an energy function;

[0123] The optimization submodule is used to iteratively optimize the stereoscopic vision interaction model using the optimization parameters until the optimization parameters reach a minimum value, and the iterative process ends.

[0124] In some optional embodiments, the convolutional encoding network includes a convolutional layer, a batch normalization layer, and a ReLU layer. The input of the convolutional decoding network is a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object. The output is a first image feature and a second image feature. The first image feature and the second image feature serve as appearance representations of the original video image in a stereoscopic visual space.

[0125] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0126] In this embodiment, the model building device for multi-target object interaction and the interaction relationship prediction device based on multi-target object interaction are presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0127] An embodiment of the present invention further provides a computer device having the above-mentioned apparatus.

[0128] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0129] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0130] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0131] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0133] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0134] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0135] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0136] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A model construction method based on multi-target object interaction, characterized in that: The method comprises: Based on the original video image, a first image feature corresponding to a first target object and a second image feature corresponding to a second target object are acquired; wherein the first target object is a flexible target object with life, and the second target object is a rigid target object without life; Acquire a first position of the first target object in the stereoscopic visual space, a second position of the second target object in the stereoscopic visual space, and an interaction position of the first target object and the second target object; Based on the first image feature and the second image feature, obtaining first distance information between the first position and the interaction position, second distance information between the second position and the interaction position, first posture information of the active organ of the first target object, and second posture information of the second target object; A stereoscopic visual interaction model of the first target object interacting with the first target object is constructed based on the first distance information, the second distance information, the first posture information, and the second posture information.

2. The method according to claim 1, characterized in that Based on the original video image, obtaining a first image feature corresponding to the first target object and a second image feature corresponding to the second target object includes: Acquiring the original video image; Processing the original video image to obtain a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object; The first image feature is extracted from the two-dimensional mask corresponding to the first target object, and the second image feature is extracted from the two-dimensional mask corresponding to the first target object through a convolutional coding network.

3. The method according to claim 1, characterized in that: Based on the first image feature and the second image feature, obtaining first distance information between the first target object and a target interaction point, second distance information between the second target object and the target interaction point, first posture information of an active organ of the first target object, and second posture information of the second target object, includes: Based on the first image feature and the second image feature, obtaining first distance information between the first target object and the target interaction point through a first decoding network; Based on the first image feature and the second image feature, obtaining second distance information between the second target object and the target interaction point through a second decoding network; Based on the first image feature and the second image feature, obtaining first posture information of the active organ of the first target object through a third decoding network; Based on the first image feature and the second image feature, second posture information of the second target object is acquired through a fourth decoding network.

4. The method according to claim 1, characterized in that Based on the first distance information, the second distance information, the first posture information, and the second posture information, a stereoscopic visual interaction model of the first target object interacting with the first target object is constructed, including: Inputting the first distance information, the second distance information, the first posture information and the second posture information into an initial stereoscopic vision interaction model for training; combining the first distance information, the second distance information, the first posture information, and the second posture information using an energy function; The stereoscopic vision interaction model is iteratively optimized using the optimization parameters until the optimization parameters reach a minimum value, and the iterative process ends.

5. The method according to claim 2, characterized in that: The convolutional encoding network includes a convolutional layer, a batch normalization layer, and a ReLU layer. The input of the convolutional decoding network is a two-dimensional mask corresponding to the first target object and a two-dimensional mask corresponding to the second target object, and the output is the first image feature and the second image feature. The first image feature and the second image feature serve as appearance representations of the original video image in a stereoscopic visual space.

6. A method for predicting interactive relationships based on multi-target object interactions, characterized in that: The method comprises: Based on the video image to be tested, obtaining a third image feature corresponding to the first object to be tested and a fourth image feature corresponding to the second object to be tested; wherein the first object to be tested is a flexible target object with life, and the second object to be tested is a rigid target object without life; Acquire a third position of the first object to be measured in the stereoscopic visual space, a fourth position of the second object to be measured in the stereoscopic visual space, and an interaction position of the first object to be measured and the second object to be measured; Based on the third image feature and the fourth image feature, obtaining third distance information between the third position and the interaction position, fourth distance information between the fourth position and the interaction position, third posture information of the active organ of the first object to be measured, and fourth posture information of the second object to be measured; The third distance information, the fourth distance information, the third posture information, and the fourth posture information are input into a trained stereoscopic vision interaction model to predict whether there is an interaction relationship between the first object to be measured and the second object to be measured.

7. A model building device for multi-target object interaction, characterized in that: The device comprises: An image feature acquisition module, used to acquire a first image feature corresponding to a first target object and a second image feature corresponding to a second target object based on an original video image; wherein the first target object is a flexible target object with life, and the second target object is a rigid target object without life; A spatial position acquisition module, used to acquire a first position of the first target object in a stereoscopic visual space, a second position of the second target object in the stereoscopic visual space, and an interaction position of the first target object and the second target object; an interaction feature acquisition module, configured to acquire, based on the first image feature and the second image feature, first distance information between the first position and the interaction position, second distance information between the second position and the interaction position, first posture information of the active organ of the first target object, and second posture information of the second target object; An interaction model construction module is used to construct a stereoscopic visual interaction model of the interaction between the first target object and the first target object based on the first distance information, the second distance information, the first posture information and the second posture information.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model building method based on multi-target object interaction described in any one of claims 1 to 5, or the interaction relationship prediction method based on multi-target object interaction described in claim 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the model building method based on multi-target object interaction described in any one of claims 1 to 6, or the interaction relationship prediction method based on multi-target object interaction described in claim 7.

10. A computer program product, characterized in that It comprises computer instructions, which are used to enable a computer to execute the model building method based on multi-target object interaction described in any one of claims 1 to 6, or the interaction relationship prediction method based on multi-target object interaction described in claim 7.

Images