Recognition system for sensing 3D scene information based on visual signals

Abstract

The invention discloses a recognition system for sensing 3D scene information based on visual signals, which comprises a picture acquisition module, a picture processing module, an operation module, a data summarization module and an output module, and is characterized in that the picture acquisition module is used for acquiring a target picture, the picture processing module is used for processing the target picture and outputting a processing result, and the data summarization module is used for outputting the processing result. The image processing module is used for processing a processing result, the operation module is used for performing operation based on the processing result and outputting an operation result, the data summarization module is used for performing data summarization on the processing result output by the image processing module and the operation result output by the operation module, and the output module is used for sorting the data summarized by the data summarization module and outputting the sorted data. The intelligent agent can efficiently and accurately perceive information such as space, people and objects in real time, so that the intelligent agent can better understand the environment and plan tasks, and the accuracy of identifying 3D scene information by the intelligent agent is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a recognition system for perceiving 3D scene information based on visual signals. Background Art

[0002] With the continuous development of computer vision and graphics technologies, 3D scene interpretation and recognition have been widely applied in various fields. Currently, 3D scene interpretation tasks include: perceiving room information, including size, orientation, and room type (bedroom, bathroom, living room, etc.); perceiving object information, including object category, size, position, orientation, color, and shape; and perceiving human information, including 2D skeleton points, 3D skeleton points, motion, and facial identification.

[0003] However, existing technologies lack recognition systems that can handle multiple tasks simultaneously and are unable to simultaneously interpret information about rooms, objects, and people. This results in the inability of intelligent agents to accurately identify scene information in 3D scenes, resulting in low recognition accuracy. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one objective of the present invention is to provide a recognition system for 3D scene information based on visual signals. This system enables an intelligent agent to accurately and efficiently perceive space, people, objects, and other information in real time, thereby facilitating the intelligent agent's better understanding of the environment and task planning, and improving the accuracy of the intelligent agent's recognition of 3D scene information.

[0005] To this end, a second object of the present invention is to provide an electronic device.

[0006] To achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention discloses a recognition system for perceiving 3D scene information based on visual signals, comprising: an image acquisition module for acquiring a target image; an image processing module connected to the image acquisition module for processing the target image and outputting a processing result; an operation module connected to the image processing module for performing an operation based on the processing result and outputting an operation result, wherein the operation result includes at least two of human body information, object information and spatial information; a data aggregation module connected to the image processing module and the operation module for summarizing the processing result output by the image processing module and the operation result output by the operation module; and an output module connected to the data aggregation module for organizing the data aggregated by the data aggregation module and outputting the organized data.

[0007] According to the embodiment of the present invention, the recognition system based on visual signal perception of 3D scene information realizes real-time, efficient and accurate perception of space, people, objects and other information by the intelligent agent through processing and calculation of the target image obtained by the acquisition module, thereby facilitating the intelligent agent to better understand the environment and plan tasks, and improving the accuracy of the intelligent agent's recognition of 3D scene information.

[0008] In addition, the recognition system based on visual signal perception of 3D scene information according to the above embodiment of the present invention may also have the following additional technical features:

[0009] In some examples, the image processing module includes: an image detection unit, connected to the image acquisition module, for detecting the target object in the target image and outputting the detection result; wherein the target object includes a human body and / or an object; a multi-target tracking unit, connected to the image detection unit, for tracking the target object in the target image based on the detection result and outputting the tracking result.

[0010] In some examples, the image processing module includes: an image segmentation unit, connected to the image acquisition module, for performing instance segmentation on the target image and outputting the segmentation results; a target extraction unit, connected to the multi-target tracking unit and the image segmentation unit, for filtering out changed target objects based on the tracking results and the segmentation results, and outputting instance information of the changed target objects.

[0011] In some examples, the operation module includes: a 2D information parsing unit, connected to the target extraction unit, for parsing the instance information of the human body and outputting the 2D bone point information of the human body based on the output of the target extraction unit when it is determined that the changed target object includes a human body.

[0012] In some examples, the operation module also includes: a 3D information prediction unit, which is respectively connected to the 2D information analysis unit and the target extraction unit, and is used to, based on the output of the target extraction unit, when it is determined that the changed target object includes a human body, perform 3D skeleton point prediction on the human body based on the instance information of the human body and the 2D skeleton point information of the human body, and output the 3D skeleton point information of the human body.

[0013] In some examples, the operation module further includes: an interactive relationship operation unit, which is connected to the 2D information parsing unit and the target extraction unit, respectively, and is used to fuse the 2D skeleton point information parsed by the 2D information parsing unit and the instance information of the changed object output by the target extraction unit, and output the interactive relationship information between the human body and the object.

[0014] In some examples, the operation module further includes: a coordinate calculation unit, connected to the target extraction unit, for calculating the 3D world coordinates of the changed target object based on the internal and external parameters of the device generating the target image and the output result of the target extraction unit.

[0015] In some examples, the operation module further includes: an object surface prediction unit, which is connected to the image acquisition module and the target extraction unit respectively, and the object surface prediction unit is used to predict the surface information of the object contained in the output results of the image acquisition module and the target extraction unit through a preset operation function.

[0016] In some examples, the operation module further includes: an attribute prediction unit, which is connected to the image acquisition module and the target extraction unit respectively, and is used to process and recognize the output results of the image acquisition module and the target extraction unit to predict multiple attributes of the objects contained therein.

[0017] In some examples, the output module includes: an adjustment unit for fine-tuning the 3D world coordinates of the object based on the data summarized by the data summary module and in combination with physical common sense, so that it satisfies the positional relationship of preset physical common sense.

[0018] In some examples, the preset physical common sense includes a support relationship, a containment relationship, or a collision relationship.

[0019] In some examples, the output module further includes: a display unit configured to display in real time the result of the fine-tuning of the 3D world coordinates of the object by the adjustment unit.

[0020] In some examples, the output module further includes: an interpretation unit configured to interpret the result of fine-tuning the 3D world coordinates of the object by the adjustment unit, and output target 3D scene information.

[0021] In some examples, the target 3D scene information includes relative positional relationships between objects.

[0022] In some examples, the recognition system for perceiving 3D scene information based on visual signals further includes: a spatial layout estimation module, which is connected to the image acquisition module and the aggregation module respectively, and is used to estimate the space in the target image based on the output of the image acquisition module using a preset spatial information estimation algorithm to obtain the spatial layout information in the target image, and transmit the spatial layout information to the data aggregation module for aggregation.

[0023] In some examples, the adjustment unit is further connected to the spatial layout estimation module to adjust the spatial layout information.

[0024] In some examples, when acquiring a target image, the image acquisition module is specifically used to: acquire the target image from a video stream, wherein the video stream includes one or more frames of images.

[0025] In some examples, the picture acquisition module is specifically used to: when the video stream includes a frame of picture, use the picture as the target picture; when the video stream includes multiple frames of pictures, extract key frame pictures from the multiple frames of pictures as the target picture.

[0026] In some examples, when the image acquisition module extracts a key frame image from the multiple frames of images as the target image, it is specifically used to: for the current frame image, when the difference in its picture content relative to the adjacent previous frame image exceeds a preset difference value, use the current frame image as the key frame image.

[0027] In some examples, the recognition system for perceiving 3D scene information based on visual signals further includes: a storage module, wherein a predefined communication specification set is stored in the storage module, and the communication specification set includes a plurality of scene information sets related to visual perception technology.

[0028] In some examples, the recognition system for perceiving 3D scene information based on visual signals further includes: an updating module connected to the storage module, for updating the communication specification set.

[0029] To achieve the above-mentioned purpose, an embodiment of a second aspect of the present invention discloses an electronic device, comprising: a recognition system for perceiving 3D scene information based on visual signals as described in the embodiment of the first aspect of the present invention.

[0030] According to the electronic device of an embodiment of the present invention, by processing and calculating the target image obtained by the acquisition module, the intelligent agent can accurately perceive the space, people, objects and other information in real time and efficiently, thereby helping the intelligent agent to better understand the environment and plan tasks, and improve the accuracy of the intelligent agent's recognition of 3D scene information.

[0031] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0033] Figure 1 is a structural block diagram of a recognition system for perceiving 3D scene information based on visual signals according to an embodiment of the present invention;

[0034] Figure 2 is a structural block diagram of a recognition system for perceiving 3D scene information based on visual signals according to another embodiment of the present invention;

[0035] Figure 3 is a structural block diagram of a recognition system for perceiving 3D scene information based on visual signals according to another embodiment of the present invention;

[0036] Figure 4 3D scene information recognition system based on visual signal perception according to another embodiment of the present invention.

[0037] Reference numerals:

[0038] Recognition system 100 based on visual signal perception of 3D scene information; image acquisition module 110; image processing module 120; image detection unit 121; multi-target tracking unit 122; image segmentation unit 123; target extraction unit 124; operation module 130; 2D information analysis unit 131; 3D information prediction unit 132; interaction relationship operation unit 133; coordinate calculation unit 134; object surface prediction unit 135; attribute prediction unit 136; data aggregation module 140; output module 150; adjustment unit 151; display unit 152; interpretation unit 153; spatial layout estimation module 160; storage module 170; update module 180. DETAILED DESCRIPTION

[0039] The embodiments of the present invention will be described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention will be described in detail below.

[0040] Reference below Figures 1-4 A recognition system 100 for perceiving 3D scene information based on visual signals according to an embodiment of the present invention is described.

[0041] Figure 1 FIG. 1 is a block diagram of a recognition system 100 for perceiving 3D scene information based on visual signals according to an embodiment of the present invention. Figure 1 As shown, the recognition system 100 for perceiving 3D scene information based on visual signals includes: an image acquisition module 110 , an image processing module 120 , a calculation module 130 , a data aggregation module 140 and an output module 150 .

[0042] The image acquisition module 110 is used to acquire a target image.

[0043] Specifically, during the recognition process of 3D scene information perceived based on visual signals, a target image can be acquired through the image acquisition module 110. Specifically, the image acquisition module 110 can interface with various image acquisition devices, such as webcams, to acquire and select target images from video streams and images captured by the webcams. It is understood that video streams include, but are not limited to, game scenes, movies, or video clips.

[0044] The image processing module 120 is connected to the image acquisition module 110 and is used to process the target image and output the processing result.

[0045] Specifically, the image processing module 120 is connected to the image acquisition module 110 and is used to process the acquired target image, including performing operations such as denoising, enhancement, and correction on the target image using preset image processing algorithms. For example, image denoising techniques can be used to reduce random noise in the image, and image enhancement techniques can be used to increase the image's contrast and brightness. Furthermore, after processing the target image, the processing results can be output to facilitate the recognition system to better identify objects and environments within the 3D scene.

[0046] The operation module 130 is connected to the image processing module 120 and is used to perform operations based on the processing results and output the operation results, which include at least two of human body information, object information and space information.

[0047] Specifically, the operation module 130 is connected to the image processing module 120, and can receive the processing results from the image processing module 120, and perform operations and analysis based on the processing results to obtain operation results. Specifically, the operation module 130 can obtain human body information from the processed image based on algorithms such as human body posture estimation and human body tracking, including the position, posture, movement, etc. of the human body. Furthermore, the operation module 130 can also obtain object information in the image from the processed image based on object detection and recognition algorithms, such as target detection models in deep learning, including furniture, home appliances, books, etc., as well as the category, position, shape, size, etc. of the object; further, the operation module 130 can also obtain spatial information in the image from the processed image based on a topological structure model of the space, including the size and shape of the space.

[0048] The data aggregation module 140 is connected to the image processing module 120 and the operation module 130 , and is used to aggregate the processing results output by the image processing module 120 and the operation results output by the operation module 130 .

[0049] Specifically, data aggregation module 140 is connected to image processing module 120 and computing module 130, and can receive and aggregate the processing results from image processing module 120 and the computation results from computing module 130. Specifically, data aggregation module 140 can integrate image data that has undergone operations such as denoising, enhancement, and correction with human body information, object information, and spatial information into a unified data set, merging the received information to obtain complete and accurate scene information.

[0050] In a specific embodiment, the data summary module 140 adds attributes to the human body information and object information based on the processing results and the calculation results. After receiving the output results, the data summary module can summarize all the attributes so that a certain data set contains the attributes newly added by all modules.

[0051] The output module 150 is connected to the data aggregation module 140 and is used to organize the data aggregated by the data aggregation module 140 and output the organized data.

[0052] Specifically, the output module 150 is connected to the data aggregation module 140, can receive the data aggregated by the data aggregation module 140, and further organize and output it, including but not limited to generating a visual representation of the 3D scene, such as a 3D model, a rendered image or a virtual reality scene, and providing a human-computer interaction interface.

[0053] Therefore, the above-mentioned recognition system 100 based on visual signal perception of 3D scene information realizes the real-time, efficient and accurate perception of space, people, objects and other information by the intelligent agent through processing and calculation of the target image obtained by the acquisition module, which is conducive to the intelligent agent to better understand the environment, plan tasks, and improve the accuracy of the intelligent agent's recognition of 3D scene information.

[0054] In a specific embodiment, the above-mentioned visual signal includes but is not limited to a third-perspective visual signal, and the above-mentioned 3D scene information includes but is not limited to 3D indoor scene information.

[0055] In one embodiment of the present invention, Figure 2 As shown, the image processing module 120 includes: an image detection unit 121, which is connected to the image acquisition module 110, and is used to detect the target object in the target image and output the detection result; wherein the target object includes a human body and / or an object; a multi-target tracking unit 122, which is connected to the image detection unit 121, and is used to track the target object in the target image based on the detection result and output the tracking result.

[0056] Specifically, the image processing module 120 includes an image detection unit 121 and a multi-target tracking unit 122, wherein the image detection unit 121 is connected to the image acquisition module 110, can receive the target image, and detect people and objects in the target image through a preset target detection algorithm, such as CNN (Convolutional Neural Networks), etc., and output the detection results; further, the multi-target tracking unit 122 is connected to the image detection unit 121, and can continuously track the target object in the target image based on the detection results, and output the tracking results to ensure that the target object does not change. For example, if occlusion or loss of field of view causes the target object ID to be inconsistent, each target object can be tracked to ensure that the target object ID remains consistent.

[0057] In one embodiment of the present invention, Figure 2 As shown, the image processing module 120 also includes: an image segmentation unit 123, which is connected to the image acquisition module 110, and is used to perform instance segmentation on the target image and output the segmentation result; a target extraction unit 124, which is connected to the multi-target tracking unit 122 and the image segmentation unit 123, and is used to filter out the changed target objects based on the tracking results and the segmentation results, and output the instance information of the changed target objects.

[0058] Specifically, the image processing module 120 also includes an image segmentation unit 123 and a target extraction unit 124, wherein the image segmentation unit 123 is connected to the image acquisition module 110, can receive the target image, and through a preset segmentation model, such as Mask R-CNN (Mask Region-Based Convolutional Neural Network, a convolutional neural network based on mask regions), perform instance segmentation on the target image and output the segmentation result; further, the target extraction unit 124 is connected to the multi-target tracking unit 122 and the image segmentation unit 123, can filter out the changed target objects based on the tracking results and segmentation results, re-interpret the changed targets, and output the instance information of the changed target objects to the downstream module so that it can be interpreted again and obtain the latest recognition results at the current moment, such as the target object's ID, location, change type (such as new addition, disappearance, deformation, etc.). In a specific embodiment, the instance information output by the target extraction unit 124 can be a dictionary, for example:

[0059]

[0060] Among them, obj represents the tracking result. Each tracking result is accompanied by an attribute. The downstream module can use this attribute to decide whether to re-interpret the target object. If it is true, re-interpretation is performed; otherwise, no interpretation is performed; bbox represents the detection box output by the image detection unit 121; rdf:type represents the category of the target object (cup / person, etc.); obj_feat represents the visual features of the target object output by the image detection unit 121, which can be a 512-dimensional feature vector; seg_countor represents the outsourced point set output by the image segmentation unit 123. Furthermore, if the tracking result is aligned with the ID, in order to ensure that the target objects at different times, that is, the same object or person, have the same ID, then after the ID is aligned, the difference in the segmentation results of the target object at time t-1 and time t can be used to determine whether there has been a change. For example, if the target object is in the first position at time t-1 and changes to the second position at time t, and the first and second positions are different and the difference is large, then it can be determined that the target object has changed.

[0061] In one embodiment of the present invention, Figure 3 As shown, the operation module 130 includes: a 2D information parsing unit 131, which is connected to the target extraction unit 124, and is used to parse the instance information of the human body according to the output of the target extraction unit 124, and output the 2D bone point information of the human body when it is determined that the changed target object includes the human body.

[0062] Specifically, the operation module 130 includes a 2D information parsing unit 131 connected to the target extraction unit 124, which is used to receive the instance information of the changed target object output by the target extraction unit 124 and make a judgment. If it is judged that the changed target object includes a human body, the 2D information parsing unit 131 can parse the instance information of the human body through a preset model and convert it into 2D bone point information of the human body for output.

[0063] In a specific embodiment, an IPR (Integral Pose Regression) model can be used to parse the instance information of the human body. The main principle is to transform the normalized heat map technology so that the value of each pixel in the normalized heat map represents the probability that the pixel point corresponds to a bone point.

[0064] In one embodiment of the present invention, Figure 3 As shown, the operation module 130 also includes: a 3D information prediction unit 132, which is respectively connected to the 2D information analysis unit 131 and the target extraction unit 124, and is used to predict the 3D skeleton points of the human body based on the instance information of the human body and the 2D skeleton point information of the human body, and output the 3D skeleton point information of the human body based on the output of the target extraction unit 124 when it is determined that the changed target object includes the human body.

[0065] Specifically, the operation module 130 also includes a 3D information prediction unit 132 connected to the 2D information analysis unit 131 and the target extraction unit 124, which is used to obtain instance information of the changed target object from the target extraction unit 124 and make a judgment. If it is judged that the changed target object includes a human body, the 3D information prediction unit 132 can obtain the 2D bone point information of the human body from the 2D information analysis unit 131, and perform 3D bone point prediction on the human body in a model preset with the 2D bone point information input value, thereby obtaining the 3D bone point information of the human body for output.

[0066] In a specific embodiment, the 2D human body posture output (ie, 2D skeleton point information) can be input into the Poselifter network model to perform 3D skeleton point prediction on the human body to obtain the 3D skeleton point information.

[0067] In one embodiment of the present invention, Figure 3 As shown, the operation module 130 also includes: an interactive relationship operation unit 133, which is connected to the 2D information parsing unit 131 and the target extraction unit 124 respectively, and is used to fuse the 2D skeleton point information parsed by the 2D information parsing unit 131 and the instance information of the changed object output by the target extraction unit 124, and output the interactive relationship information between the human body and the object.

[0068] Specifically, the operation module 130 also includes an interaction relationship operation unit 133 connected to the 2D information parsing unit 131 and the target extraction unit 124, which can fuse the 2D skeleton point information and the instance information of the changed object, including but not limited to combining the posture information of the human body with the position and motion information of the object to analyze the possible interaction relationship between them, etc. For example, if the hand skeleton point of the human body in the target object is close to an object and the position of the object has changed, it means that the human body is operating or moving the object. Further, the interaction relationship operation unit 133 can output the interaction relationship information between the human body and the object, including but not limited to displaying it in a visual manner, such as an interaction diagram, a trajectory diagram, etc. It can be understood that the interaction relationship operation unit 133 can use the human body, the object and the characteristics of the human body to associate the interaction between the human body and the object, thereby realizing the classification of actions in images or videos.

[0069] In a specific embodiment, the idea of ​​PMN (Pose-based Modular Network for Human-Object Interaction Detection) model can be used to build a formula model based on the relationship between people and objects, for example as well as in, Represents the relative spatial posture feature of the i-th skeleton point, The relative spatial posture feature of the i-th bone point is composed of two values, Represents the first value, Indicates the second value, W represents the width of the image, such as 1280, H represents the height of the image, such as 720, x i Represents the x-axis coordinate of the i-th bone point, Indicates the center point of the object in the x direction, y i Indicates the y-axis coordinate of the i-th bone point, represents the center point of the item in the y direction, and represents the absolute posture feature of the i-th skeleton point, The absolute posture feature of the i-th bone point consists of two values, Indicates the center point of the human detection frame in the x-axis direction, The center point of the y-axis direction of the detection frame of the person, x i Indicates the first value, y〃 i Indicates the second value.

[0070] Furthermore, the existing image visual information, 2D spatial information, and newly added human skeleton point information can be fused to obtain the probability distribution of interaction relationship information. When the probability of interaction relationship information exceeds a preset threshold, it means that a certain interaction relationship has occurred.

[0071] In one embodiment of the present invention, Figure 3 As shown, the operation module 130 also includes: a coordinate calculation unit 134, which is connected to the target extraction unit 124 and is used to calculate the 3D world coordinates of the changed target object based on the internal and external parameters of the target image generation device and the output result of the target extraction unit 124.

[0072] Specifically, the operation module 130 also includes a coordinate calculation unit 134 connected to the target extraction unit 124, which is used to perform a coordinate conversion operation by receiving the internal and external parameters of the target image generation device and the target extraction result, converting the target object from the image coordinate system to the three-dimensional world coordinate system, and calculating the 3D world coordinates of the changed target object. It can be understood that the intrinsic parameters of the generation device are hardware properties of the generation device and can be directly obtained, while the extrinsic parameters can be calibrated through the CNN model, that is, a given target image can be input into a trained model to obtain the corresponding extrinsic parameters.

[0073] In one embodiment of the present invention, Figure 3As shown, the operation module 130 also includes: an object surface prediction unit 135, which is connected to the image acquisition module 110 and the target extraction unit 124 respectively. The object surface prediction unit 135 is used to predict the surface information of the object contained in the output results of the image acquisition module 110 and the target extraction unit 124 through a preset operation function.

[0074] Specifically, the operation module 130 also includes an object surface prediction unit 135 connected to the image acquisition module 110 and the target extraction unit 124, which is used to receive the object in the image acquisition module 110 and the object information in the output results of the target extraction unit 124, including but not limited to the bounding box of the object, and predict the surface information of the object, such as material type, color distribution, etc., through a preset operation function.

[0075] In a specific embodiment, the LDIF (Local Deep Implicit Functions) model can be used. Its main principle is to determine whether a point is inside or outside the surface of an object. It mainly includes two functions: 1) the g function, which is composed of N basic shape elements. Each shape element represents an area on the surface of an object and can describe the general shape of a basic element and the relationship between adjacent basic elements; 2) the f function, which can obtain more detailed geometric information for each basic shape element. The LDIF function can be used to determine whether a point in space is inside or outside the object, that is, to determine a boundary, which is the surface of the object. Furthermore, combined with the MC (Marching Cube) algorithm, a representation of the object surface information that can be used by a computer can be constructed.

[0076] In one embodiment of the present invention, Figure 3 As shown, the operation module 130 also includes: an attribute prediction unit 136, which is connected to the image acquisition module 110 and the target extraction unit 124 respectively, and is used to process and recognize the output results of the image acquisition module 110 and the target extraction unit 124 to predict various attributes of the objects contained therein.

[0077] Specifically, the operation module 130 also includes an attribute prediction unit 136 connected to the image acquisition module 110 and the target extraction unit 124, respectively, for receiving the target image of the image acquisition module 110 and the output result of the target extraction unit 124, and processing and recognizing them through a preset model to predict the attributes of the objects contained therein, such as color, shape, etc. Specifically, the attribute prediction unit 136 can traverse the output results of the image acquisition module 110 and the target extraction unit 124. If the instruction of the output result is True, then the attribute prediction unit 136 can extract the image of the object based on the bounding box and input it into the classification model to identify the color, shape, etc. of the object, and label each object with multiple corresponding attribute labels.

[0078] In one embodiment of the present invention, Figure 4 As shown, the output module 150 includes: an adjustment unit 151 for fine-tuning the 3D world coordinates of the object based on the data summarized by the data summary module 140 and in combination with physical common sense, so that it meets the positional relationship of preset physical common sense.

[0079] Specifically, the output module 150 includes an adjustment unit 151, which is used to receive the data summarized by the data summary module 140 and fine-tune the 3D world coordinates of the object based on physical common sense. For example, there should be a certain distance between two objects, but the calculated coordinates show that they are too close or overlapping. The adjustment unit 151 can adjust the 3D world coordinates of the object to reflect the actual situation so that it meets the positional relationship of the preset physical common sense.

[0080] In a specific embodiment, based on common sense in physics, the fine-tuning of the 3D world coordinates of the object is mainly based on the following formula:

[0081]

[0082]

[0083] as well as Among them, ε s (pg) represents the energy function value of the support relationship of a specific pg,

[0084] v i Indicates the identifier of object i, which is the supported object, such as an apple, v j represents the representation of object j, here it refers to the supporting object, such as a table, ε s , ε o , ε height They represent an energy function, s represents support, o represents overlap, height represents the height between objects, ε o (v i ,v j) represents the specific energy function value, which is v i and v j The overlap rate of the detection frames of two objects on the horizontal plane, ε height (v i ,v j ) represents the specific energy function value, which is v i The lower surface and v j The height difference of the upper surface, v represents an object or a person, V object Represents a collection of objects, V human Represents a collection of people, V layout Represents the set of spaces, V containe represents a collection of containers, c represents the intersection between entities, the volume where the detection boxes of two items overlap, c(v,V layout ) represents the volume of objects or people inserted into the ground / wall / ceiling, c(v i ,v j ) represents the volume of objects and people interspersed, E hoio represents the set of character interactions (objects, people). For example, if a character interaction is holding a pen, the set includes a (hand, pen). ε(I|pg) represents the likelihood function of the image and the prediction result (analysis graph). I represents the current image, and pg represents the prediction result, including the skeleton point of the person, the detection frame of the object, etc. o represents the overlapping area of ​​the detection box and the projected polygon, B(v) represents the detection box of object v, Represents the polygon of the 3D detection box of object v projected onto the image, D h Represents the Euclidean distance between the 2D detection result and the 3D projection result, Po(v) represents the 2D bone point detection result of a person, The result of projecting a person's 3D skeleton points onto the image.

[0085] In one embodiment of the present invention, the preset physical common sense includes a support relationship, a containment relationship, or a collision relationship.

[0086] Specifically, the preset physical common sense includes support relationship, containment relationship or collision relationship, among which, the support relationship can be a relationship in which one object supports or depends on another object, which is usually manifested in 3D scenes as one object being above or below another object. For example, a person stands on the ground, and the ground provides support. A table is placed on the ground, and there is also a support relationship between the table and the ground; the containment relationship can be an object being located inside another object, which is usually manifested in 3D scenes as the boundary of one object surrounding the boundary of another object. For example, a person stands in a room, and the person is contained by the room. An object is wrapped by another object, which can also form a containment relationship; the collision relationship can be a situation in which two objects come into contact in space, which is usually manifested in 3D scenes as the boundary of one object touching the boundary of another object. For example, a person stands on the ground, in contact with the ground, and there is no obvious gap, so the ground and the person form a collision relationship.

[0087] In one embodiment of the present invention, Figure 4 As shown, the output module 150 further includes: a display unit 152 for displaying in real time the result of the fine-tuning of the 3D world coordinates of the object by the adjustment unit 151 .

[0088] Specifically, the output module 150 further includes a display unit 152 for updating and displaying, in real time, the results of fine-tuning the object's 3D world coordinates by the adjustment unit 151, and presenting them to the user in a visual form. Specifically, the display unit 152 can present the adjusted 3D world coordinates in a graphical manner, including rendering the object model in the 3D scene, drawing the object's bounding box, using color to distinguish different objects, etc.

[0089] In one embodiment of the present invention, Figure 4 As shown, the output module 150 further includes: an interpretation unit 153, which is used to interpret the result of fine-tuning the 3D world coordinates of the object by the adjustment unit 151, and output the target 3D scene information.

[0090] Specifically, the output module 150 also includes an interpretation unit 153, which is used to receive the fine-tuned 3D world coordinate data from the adjustment unit 151 for interpretation, and output the target 3D scene information in the form of text, graphics or data structure, including but not limited to the name, attributes, position, posture, etc. of each object in the 3D scene.

[0091] In one embodiment of the present invention, the target 3D scene information includes the relative position relationship between objects.

[0092] Specifically, the target 3D scene information includes the relative position relationship between objects. The relative position relationship can describe the spatial layout and relative position between different objects in the scene. For example, A isOn B means that A falls on B, and three conditions must be met at the same time: the lowest point of A is not lower than the highest point of B, the center point of A is within the upper surface range of B, otherwise it cannot be supported, and the difference between the lowest point of A and the highest point of B does not exceed a certain threshold, otherwise A is suspended above B; A isIn B means that A is within B, and two conditions must be met at the same time: the center point of A is within the 3D bounding box of B, and the bounding box of A is not larger than the bounding box of B.

[0093] In one embodiment of the present invention, the recognition system 100 for perceiving 3D scene information based on visual signals further includes: a spatial layout estimation module 160, which is connected to the image acquisition module 110 and the aggregation module respectively, and is used to estimate information on the space in the target image based on the output of the image acquisition module 110 using a preset spatial information estimation algorithm, obtain spatial layout information in the target image, and transmit the spatial layout information to the data aggregation module 140 for aggregation.

[0094] Specifically, the spatial layout estimation module 160 is connected to the image acquisition module 110 and the aggregation module respectively, and can use a preset spatial information estimation algorithm to estimate the information of the space in the target image output by the acquisition module to obtain the spatial layout information in the target image and transmit it to the data aggregation module 140 for aggregation, wherein the spatial layout information includes but is not limited to the size, shape, position, placement and relative relationship of the space.

[0095] In a specific embodiment, when an intelligent agent enters a space and looks around, it can generate point cloud data of the space. Based on this data, an algorithm for plane extraction using aggregate clustering can be used to detect planes. After filtering out walls, floors, and ceilings through a rule-based machine learning algorithm, the three-dimensional bounding box of the entire space, that is, the spatial layout, can be estimated.

[0096] In one embodiment of the present invention, the adjustment unit 151 is further connected to the space layout estimation module 160 for adjusting the space layout information.

[0097] Specifically, adjustment unit 151 is also connected to spatial layout estimation module 160 and can adjust and optimize errors in spatial layout information using a pre-set algorithm to improve the accuracy of spatial layout information. Specifically, adjustment unit 151 can enhance detailed spatial distribution information that spatial layout estimation module 160 cannot capture, such as the precise position and shape of objects, making spatial layout estimation module 160's perception of the 3D scene more comprehensive and detailed. Furthermore, adjustment unit 151 can also optimize the relative positions of objects in the spatial layout information to ensure that they conform to pre-set physical common sense.

[0098] In one embodiment of the present invention, when acquiring a target image, the image acquisition module 110 is specifically configured to: acquire the target image from a video stream, wherein the video stream includes one or more frames of images.

[0099] Specifically, the image acquisition module 110 can obtain the target image from the video stream. The image acquisition module 110 can establish a connection with a video source, such as a camera, a video file, a network video stream, etc., and stably extract single-frame or multi-frame images from the video stream according to a preset algorithm.

[0100] In one embodiment of the present invention, the picture acquisition module 110 is specifically configured to: when the video stream includes one frame of picture, use the picture as the target picture; when the video stream includes multiple frames of pictures, extract key frame pictures from the multiple frames of pictures as the target picture.

[0101] Specifically, in the process of acquiring the target image, when the video stream has only one frame of image, that is, one frame of image can contain all the information of the entire 3D scene, the image acquisition module 110 can directly extract this frame of image as the target image; when the video stream includes multiple frames of image, the video stream is a dynamic sequence, and contains the changes of the 3D scene at different time points, the key frame image can be extracted from the multiple frames of image as the target image, that is, the extracted key frame image can provide representative information of the 3D scene.

[0102] In one embodiment of the present invention, when the picture acquisition module 110 extracts a key frame picture as a target picture from multiple frames of pictures, it is specifically used to: for the current frame picture, when the difference in its picture content relative to the adjacent previous frame picture exceeds a preset difference value, the current frame picture is used as the key frame picture.

[0103] Specifically, when extracting a key frame image from multiple frames as a target image, the image acquisition module 110 may compare the images in the video stream frame by frame and select a key frame image based on the difference in image content. Specifically, for a current frame image, when it is detected that the image content difference between the current frame image and the previous frame image exceeds a preset difference value, the image acquisition module 110 may select the current frame image as a key frame image.

[0104] In a specific embodiment, the preset difference value can be set based on experience, experiments, or requirements of a specific application scenario.

[0105] In one embodiment of the present invention, the recognition system 100 for perceiving 3D scene information based on visual signals further includes: a storage module 170, which stores a predefined communication specification set, and the communication specification set includes a variety of scene information sets related to visual perception technology.

[0106] Specifically, the storage module 170 is used to store a predefined communication specification set, which includes a variety of scene information sets related to visual perception technology, including but not limited to 2D skeleton point information, 3D world coordinates, and attribute information of target objects in the 3D scene. For example, 3D world coordinates can be uniformly defined as coordinateX = [2.3], coordinateY = [3.3], coordinateZ = [4.4]; 2D skeleton point information can be uniformly defined as 2D_humanpose = [[x1, y1], [x2, y2], [x3, y3] ... [xn, yn]], 2D_keypoint_name = ["leg", "hand", "arm" ... "nose"], etc.

[0107] In one embodiment of the present invention, the recognition system 100 for perceiving 3D scene information based on visual signals further includes an updating module 180 connected to the storage module 170 for updating the communication specification set.

[0108] Specifically, update module 180 is connected to storage module 170 and can update the communication specification set to ensure the real-time and accuracy of the communication specification set in storage module 170. It is understood that with technological advancements and changes in 3D scenes, update module 180 can adaptively add scene information that does not appear in the communication specification set to the communication specification set. Furthermore, update module 180 can also adjust the scene information in the communication specification set and optimize the communication specification set to adapt to new scene requirements.

[0109] In summary, according to the embodiment of the present invention, the recognition system 100 based on visual signal perception of 3D scene information realizes the real-time, efficient and accurate perception of space, people, objects and other information by the intelligent agent through processing and calculation of the target image obtained by the acquisition module, thereby facilitating the intelligent agent to better understand the environment and plan tasks, and improving the accuracy of the intelligent agent's recognition of 3D scene information. Furthermore, by decomposing the recognition system 100 based on visual signal perception of 3D scene information into multiple independent modules, each of which is responsible for processing specific tasks separately, each module can be optimized independently without affecting the performance of other modules. At the same time, the addition of the update module 180 and the storage module 170 also enables any newly added modules to be seamlessly integrated into the recognition system 100 based on visual signal perception of 3D scene information without the need for large-scale changes to the entire recognition system 100 based on visual signal perception of 3D scene information.

[0110] A further embodiment of the present invention further provides an electronic device, comprising: the recognition system 100 for perceiving 3D scene information based on visual signals as described in the embodiment of the first aspect of the present invention.

[0111] It should be noted that when the electronic device performs recognition based on 3D scene information, its specific implementation method is similar to the specific implementation method of the 3D scene information recognition system based on visual signal perception in any of the above-mentioned embodiments of the present invention. Therefore, for a detailed exemplary description of the process of the electronic device performing recognition based on 3D scene information, please refer to the aforementioned relevant description part of the 3D scene information recognition system based on visual signal perception. In order to reduce redundancy, it will not be repeated here.

[0112] According to the electronic device of the embodiment of the present invention, by processing and calculating the target image obtained by the acquisition module, the intelligent agent realizes real-time, efficient and accurate perception of space, people, objects and other information, which is conducive to the intelligent agent to better understand the environment and plan tasks, and improves the accuracy of the intelligent agent in recognizing 3D scene information. Furthermore, by decomposing the recognition system 100 based on visual signal perception of 3D scene information into multiple independent modules, each of which is responsible for processing specific tasks separately, each module can be optimized independently without affecting the performance of other modules. At the same time, the addition of the update module 180 and the storage module 170 also enables any newly added modules to be seamlessly integrated into the recognition system 100 based on visual signal perception of 3D scene information without the need for large-scale changes to the entire recognition system 100 based on visual signal perception of 3D scene information.

[0113] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0114] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A recognition system for 3D scene information based on visual signal perception, characterized in that: include: Image acquisition module, used to acquire target images; An image processing module, connected to the image acquisition module, is used to process the target image and output the processing result; a calculation module connected to the image processing module, configured to perform calculations based on the processing results and output calculation results, wherein the calculation results include at least two of human body information, object information, and spatial information; a data aggregation module, connected to the image processing module and the operation module, for aggregating the processing results output by the image processing module and the operation results output by the operation module; The output module is connected to the data aggregation module and is used to organize the data aggregated by the data aggregation module and output the organized data.

2. The recognition system for perceiving 3D scene information based on visual signals according to claim 1, characterized in that: The image processing module includes: An image detection unit, connected to the image acquisition module, configured to detect a target object in the target image and output a detection result; wherein the target object includes a human body and / or an object; A multi-target tracking unit is connected to the image detection unit and is used to track the target object in the target image based on the detection result and output a tracking result.

3. The recognition system for perceiving 3D scene information based on visual signals according to claim 2, characterized in that: The image processing module includes: An image segmentation unit, connected to the image acquisition module, configured to perform instance segmentation on the target image and output a segmentation result; The target extraction unit is connected to the multi-target tracking unit and the image segmentation unit, and is used to filter out the changed target objects based on the tracking results and the segmentation results, and output instance information of the changed target objects.

4. The recognition system for perceiving 3D scene information based on visual signals according to claim 3, characterized in that: The operation module includes: The 2D information parsing unit is connected to the target extraction unit and is used to parse the instance information of the human body and output the 2D skeleton point information of the human body according to the output of the target extraction unit when it is determined that the changed target object includes the human body.

5. The recognition system for perceiving 3D scene information based on visual signals according to claim 4, characterized in that: The operation module further includes: A 3D information prediction unit is connected to the 2D information parsing unit and the target extraction unit, respectively, and is used to predict the 3D skeleton points of the human body based on the instance information of the human body and the 2D skeleton point information of the human body, and output the 3D skeleton point information of the human body based on the output of the target extraction unit when it is determined that the changed target object includes a human body.

6. The recognition system for perceiving 3D scene information based on visual signals according to claim 5, characterized in that: The operation module further includes: The interactive relationship operation unit is connected to the 2D information parsing unit and the target extraction unit respectively, and is used to fuse the 2D skeleton point information parsed by the 2D information parsing unit and the instance information of the changed object output by the target extraction unit, and output the interactive relationship information between the human body and the object.

7. The recognition system for perceiving 3D scene information based on visual signals according to claim 2, characterized in that: The operation module further includes: A coordinate calculation unit is connected to the target extraction unit and is used to calculate the 3D world coordinates of the changed target object based on the internal and external parameters of the device generating the target image and the output result of the target extraction unit.

8. The recognition system for perceiving 3D scene information based on visual signals according to claim 7, characterized in that: The operation module further includes: An object surface prediction unit is connected to the image acquisition module and the target extraction unit respectively. The object surface prediction unit is used to predict the surface information of the object contained in the output results of the image acquisition module and the target extraction unit through a preset operation function.

9. The recognition system for perceiving 3D scene information based on visual signals according to claim 8, characterized in that: The operation module further includes: The attribute prediction unit is connected to the image acquisition module and the target extraction unit respectively, and is used to process and perform recognition operations on the output results of the image acquisition module and the target extraction unit to predict multiple attributes of the objects contained therein.

10. The recognition system for perceiving 3D scene information based on visual signals according to claim 8, characterized in that: The output module includes: The adjustment unit is used to fine-tune the 3D world coordinates of the object based on the data summarized by the data summary module and in combination with physical common sense, so that the 3D world coordinates of the object meet the positional relationship of preset physical common sense.

11. The recognition system for perceiving 3D scene information based on visual signals according to claim 10, characterized in that: The preset physical common sense includes a support relationship, a containment relationship, or a collision relationship.

12. The recognition system for perceiving 3D scene information based on visual signals according to claim 10, characterized in that: The output module also includes: The display unit is used to display in real time the result of the fine-tuning of the 3D world coordinates of the object by the adjustment unit.

13. The recognition system for perceiving 3D scene information based on visual signals according to claim 10, characterized in that: The output module also includes: The interpretation unit is used to interpret the result of fine-tuning the 3D world coordinates of the object by the adjustment unit, and output target 3D scene information.

14. The recognition system for perceiving 3D scene information based on visual signals according to claim 13, characterized in that: The target 3D scene information includes the relative position relationship between objects.

15. The recognition system for perceiving 3D scene information based on visual signals according to claim 10, characterized in that: Also includes: The spatial layout estimation module is connected to the image acquisition module and the aggregation module respectively, and is used to estimate the space in the target image based on the output of the image acquisition module using a preset spatial information estimation algorithm to obtain the spatial layout information in the target image, and transmit the spatial layout information to the data aggregation module for aggregation.

16. The recognition system for perceiving 3D scene information based on visual signals according to claim 15, characterized in that: The adjustment unit is also connected to the space layout estimation module and is used to adjust the space layout information.

17. The recognition system for perceiving 3D scene information based on visual signals according to claim 1, characterized in that: When acquiring a target image, the image acquisition module is specifically used to: The target image is obtained from a video stream, wherein the video stream includes one or more frames of images.

18. The recognition system for perceiving 3D scene information based on visual signals according to claim 17, characterized in that: The image acquisition module is specifically used to: When the video stream includes a frame of picture, taking the frame of picture as the target picture; When the video stream includes multiple frames of pictures, a key frame picture is extracted from the multiple frames of pictures as the target picture.

19. The recognition system for perceiving 3D scene information based on visual signals according to claim 18, characterized in that: When the picture acquisition module extracts a key frame picture from the multiple frames of pictures as the target picture, it is specifically used to: For a current frame picture, when the difference in picture content between the current frame picture and the adjacent previous frame picture exceeds a preset difference value, the current frame picture is used as the key frame picture.

20. The recognition system for perceiving 3D scene information based on visual signals according to any one of claims 1 to 19, characterized in that: Also includes: A storage module stores a predefined communication specification set, wherein the communication specification set includes a plurality of scene information sets related to visual perception technology.

21. The recognition system for perceiving 3D scene information based on visual signals according to claim 20, characterized in that: Also includes: An updating module is connected to the storage module and is used to update the communication specification set.

22. An electronic device, characterized in that: include: A recognition system for perceiving 3D scene information based on visual signals as described in any one of claims 1 to 21.

Images