Scene recognition method and device based on multi-source information fusion

Abstract

The invention discloses a scene recognition method and device based on multi-source information fusion, and the method comprises the steps: carrying out the preliminary classification of a detected scene through the collection data of a single-line laser radar, further classifying the detected scene through the collection data of a depth camera, and carrying out the recognition of the detected scene through combining the detection results of different sensors. And information loss among different sensors can be supplemented. When vision is influenced by light rays or visual features in the environment are few, other sensors can be kept stable, and robustness of overall recognition is guaranteed; meanwhile, an object in a detected scene is detected through a target detection model, and the semantic meaning of the detected object is deduced in combination with the association reasoning ability of a knowledge graph, so that the scene where the robot is located is deduced, a basis is provided for subsequent task decision making, and the robot can select an optimal task execution strategy according to the scene; and through comprehensive utilization of multi-dimensional information, the scene identification precision is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of environmental perception and knowledge reasoning, and specifically relates to a scene recognition method and device based on multi-source information fusion. Background Art

[0002] With the development of robot technology, autonomous mobile robots have been widely used in scenarios such as home services, logistics handling, security patrols, and emergency rescue. In these scenarios, robots need to be able to perceive the surrounding environment in real time and adjust their task execution strategies according to environmental changes, which places high demands on the robot's scene recognition and understanding capabilities. For example, a sweeping robot in the bedroom actively avoids cleaning during human rest time; stubborn stains are prone to occur in the kitchen, so a more powerful cleaning strategy is adopted; in addition, when a home service robot performs a target search task, if the target is kitchenware, it will go to the kitchen first and temporarily skip places with a low probability of appearing, such as the living room. Therefore, scene recognition plays an important role in optimizing the robot's task execution process, and can significantly improve the robot's intelligence level and task execution efficiency.

[0003] Currently, most existing scene recognition methods rely on vision, lidar or other sensors for environmental perception. However, in complex environments (such as corridors and rooms in homes), the data of a single sensor is susceptible to external interference, resulting in inaccurate or unstable scene classification results. For example, lidar may misidentify in scenes with structures similar to rooms due to similar features; visual perception may fail due to insufficient light or obstacles, or due to low image quality when the robot is moving, insufficient image feature information is obtained, resulting in recognition errors. Summary of the invention

[0004] The purpose of this invention is to provide a scene recognition method based on multi-source information fusion, which comprehensively analyzes the scene through multimodal information (including lidar point cloud data, target detection results and knowledge reasoning), thereby realizing intelligent understanding and judgment of the robot's environment.

[0005] In a first aspect, the present invention provides a scene recognition method based on multi-source information fusion, which comprises the following steps: Step 1: Use the knowledge graph embedding model to obtain the association probability p between objects and scenes; Step 2: Collect point cloud data of the scene under test; fit the collected point cloud data into multiple straight lines, and preliminarily classify the scene under test according to the number of point clouds in the fitted straight lines; Step 3: Collect images of the scene under test, use the target detection model to detect the collected images, and obtain objects in the scene under test; Step 4: Obtain the final scene probability of the current frame based on the objects in the scene being tested and the corresponding association probability p ; Using the final scene probability Further classify the tested scenarios and obtain the types of the tested scenarios; Step 5: Repeat steps 2 to 4 to obtain the type of the scene being tested in each frame of the image and complete scene recognition.

[0006] Preferably, in the step 2, the preliminary classification method is: counting the two fitting lines with the largest number of point clouds among multiple fitting lines, if the sum of the number of point clouds in these two fitting lines exceeds a preset threshold, then the measured scene is judged to be a corridor; otherwise, the measured scene is judged to be a room.

[0007] As a preferred embodiment, in the step 4, the final scene probability is used The method for further classifying the tested scene is as follows: if the tested scene is determined to be a corridor in step 2, the scene of the current frame is output as a corridor; if the tested scene is determined to be a room in step 2 and the final scene probability of the current frame is Greater than the scene probability threshold When the scene type of the current frame is output as greater than the threshold The final scenario probability The corresponding scene type.

[0008] As a preference, in step 4, the final scene probability The method of obtaining is: obtain the scene probability corresponding to each object in the current frame in turn , scenario probability The expression is: in, and is the scene probability corresponding to the previous object; is the attenuation factor; is the number of consecutive discoveries of the same object; is the association probability between the nth object and the i-th scene; is the association probability between the nth object and the jth scene; M is the number of scenes; ; ; N is the number of objects detected in the current frame; The scene probability corresponding to the last object in the current frame As the final scene probability of the i-th scene in the current frame ; Use the final scene probability of the previous frame as the scene probability of the current frame ; If the scene of the previous frame is a corridor, the probability of the final scene of the previous frame is .

[0009] Preferably, in step 2, the method of fitting the straight line is as follows: Step 1-1. Convert the collected point cloud data into Cartesian coordinate format; Step 1-2. Fit the point cloud data point by point. If the distance between the extended point and the current fitted line is less than the set range threshold, the extended point is considered to belong to the current fitted line; otherwise, the extended point is used as the starting point of the new fitted line; Step 1-3. Merge two fitted straight lines belonging to the same straight line according to the similarity of slope and intercept.

[0010] Preferably, in the step 1, the point cloud data is collected by a single-line laser radar.

[0011] Preferably, in the step 2, noise filtering is performed on the point cloud data collected by the single-line laser radar to exclude points in the point cloud data whose distance from the laser radar is less than 0.1 meters or greater than 20 meters.

[0012] Preferably, in the step one, the method for obtaining the association probability p is: constructing a knowledge graph dataset; using the knowledge graph dataset to train the knowledge graph embedding model; based on the trained knowledge graph embedding model, obtaining the association score x between the object and the scene; and obtaining the association probability p after normalizing the association score x.

[0013] Preferably, in the step 1, during the training process, the expression of the loss function L is set as follows: in, To fix the boundary threshold, control the distance between positive and negative examples; is the defined distance function; is the sigmoid function; h, r, t are the vectors corresponding to the head entity, relation, and tail entity respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector; It is Hadamard.

[0014] In the second aspect, the present invention provides a scene recognition device based on multi-source information fusion, which includes a carrying component and a single-line laser radar, a depth camera and a control module installed on the carrying component; the control module is used to execute the above-mentioned scene recognition method; the control module includes a straight line detection module and a scene probability update module; the straight line detection module is used to preliminarily classify the type of the measured scene according to the point cloud data; the scene probability update module is used to further classify the measured scene according to the image of the object.

[0015] The present invention has the following beneficial effects: 1. The present invention preliminarily classifies the measured scene through the data collected by the single-line laser radar, and further classifies the measured scene through the data collected by the depth camera. By combining the detection results of different sensors, the information missing between different sensors can be supplemented. When vision is affected by light or there are fewer visual features in the environment, other sensors can remain stable to ensure the robustness of the overall recognition.

[0016] 2. The present invention detects objects in the tested scene through the target detection model, and combines the associative reasoning ability of the knowledge graph to infer the semantics of the detected objects, and then infers the scene where the robot is located, providing a basis for subsequent task decision-making, so that the robot can select the optimal task execution strategy according to the scene; through the comprehensive use of multi-dimensional information, the scene recognition accuracy is high.

[0017] 3. The present invention can directly use the output results of the target detection algorithm to perform scene inference without the need for multiple inferences on the original image using a deep learning method, which greatly reduces the consumption of computing resources and time and is suitable for scenarios with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 Schematic diagram of a scene recognition device in the present invention.

[0020] Figure 2 It is an overall flow chart of the scene recognition method in the present invention.

[0021] Figure 3 Schematic diagram of the probability update process for different scenarios; (a) is a schematic diagram of the probability update process for the laboratory scenario; (b) is a schematic diagram of the probability update process for the bathroom scenario. DETAILED DESCRIPTION

[0022] The technical solution provided by the present invention will be described in detail below in conjunction with specific examples. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0023] like Figure 1As shown, a scene recognition device based on multi-source information fusion includes a carrying component and a single-line laser radar, a depth camera and a control module installed on the carrying component; the single-line laser radar is used to collect point cloud data of the measured scene; the depth camera is used to collect images of objects in the measured scene; the control module includes a straight line detection module and a scene probability update module; the straight line detection module is used to preliminarily classify the type of the measured scene according to the point cloud data; the scene probability update module is used to further classify the type of the measured scene according to the image of the object.

[0024] like Figure 2 As shown, a scene recognition method based on multi-source information fusion uses the above-mentioned scene recognition device, which includes the following steps: Step 1: Get the probability of association between objects in the scene and the scene 1-1. Construct a knowledge graph dataset, where the relationship attributes include object instances, object categories, and room types. The relationship attributes include at least the following two types: the relationship between the object instance and the object category (such as: belongs to), and the relationship between the object instance and the room (such as: the room it is in). In addition, some unknown entities are allowed to exist, which can be automatically completed by the model in the subsequent reasoning process. The partial knowledge graph dataset constructed in this example is shown in Table 1.

[0025] Table 1 Knowledge graph dataset 1-2. The knowledge graph is embedded through the RotateE model, and the efficiency and effectiveness of the model are improved through self-adversarial negative sampling technology. The goal is to minimize the score function of the triples so that the scores of the real triples are higher and the scores of the false triples are lower. The expression of the loss function L is:

[0026] (1) (2) in, To fix the boundary threshold, control the distance between positive and negative examples; is the defined distance function; is the sigmoid function, which is used to map the score to the range of [0,1]; h, r, t are the vectors corresponding to the head entity, relation, and tail entity respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector; It is Hadamard.

[0027] 1-3. Based on the trained RotateE model, perform tail reasoning tasks to obtain the association scores x between various objects and scenes.

[0028] 1-4. Normalize the association score x using formula (9) to construct a database of association probabilities between objects and scenes.

[0029] (3) in, is the association probability between the object and the scene.

[0030] Step 2: Preliminary classification of the types of tested scenarios 2-1. Use a single-line laser radar to collect point cloud data of the measured scene; filter the laser scanning data for noise, and exclude points in the point cloud that are less than 0.1 meters or greater than 20 meters away from the laser radar to reduce the interference of measurement noise on subsequent processing and ensure the validity of the point cloud data.

[0031] 2-2. Convert the polar coordinate data collected by the laser radar (including the angle range of the laser beam and the distance data) into the Cartesian coordinate format (x, y) of the two-dimensional point cloud, and calculate the distance r between each point and the laser radar and the corresponding angle. Calculate the position information in the Cartesian coordinate system, as shown in formula (1): Special styles(4) 2-3. Using the orderliness of the laser radar scanning output point cloud, all straight line segments are identified through the incremental straight line fitting method: starting from the starting point of the point cloud data, point by point fitting is performed, and the least squares method is used to fit the straight line to detect continuous straight line segments. The straight line point cloud cluster is ; where x i and i are the horizontal and vertical coordinates of the i-th point in the straight line segment, respectively; j is the number of points in the straight line segment; . Represent the point cloud collection in matrix form:

[0032] (5) The expression of the linear parameters fitted by the least squares algorithm is: (6) Where m is the slope, c is the intercept, and T is the transposed sign.

[0033] The point currently added to the current fitting line is taken as the extension point. If the extension point satisfies formula (4), that is, the distance between the extension point and the current fitting line is less than the set range threshold, the extension point is considered to belong to the current fitting line; otherwise, the extension point is taken as the starting point of the new fitting line.

[0034] (7) in, is the extension point coordinate; The range threshold to set.

[0035] 2-4. According to the similarity of slope and intercept, the fitted lines belonging to the same line are merged; if two fitted lines satisfy formulas (5) and (6) at the same time, the two fitted lines are considered to belong to the same line.

[0036] (8) (9) Among them, m1 and m2 are the slopes of the two straight lines; c1 and c2 are the intercepts of the two straight lines; and are the slope similarity threshold and intercept similarity threshold, respectively.

[0037] 2-5. The two fitted lines with the largest number of point clouds are counted. If the sum of the number of point clouds in these two fitted lines exceeds a preset threshold, the measured scene is judged to be a corridor; otherwise, the measured scene is judged to be a room.

[0038] In this embodiment, the horizontal field of view of the single-line laser radar is 270°, the scanning frequency is 30HZ, the number of point clouds in a single scanning cycle is 1440, the detection distance is 25m, and the corridor width is 3.72m; the point cloud number threshold is set to 300.

[0039] Step 3: Detect objects in the scene 3-1. Construct a target image dataset, which includes images of various objects and their annotation information; divide the target image dataset into training set, validation set and test set in a ratio of 7:2:1. When dividing, it is necessary to ensure that images of different scenes and categories are evenly distributed in the training set and validation set to avoid the model training being biased towards a certain type of data.

[0040] 3-2. Use the training set to train the YOLO target detection model; set the training rounds to 100 times to ensure that the model fully learns the characteristics of different scenarios; use the learning rate scheduler to dynamically adjust the learning rate during training to avoid overfitting or underfitting. Add an early stopping mechanism to dynamically terminate training based on the accuracy of the validation set to avoid overtraining and resulting in a decrease in model performance.

[0041] 3-3. Use the depth camera to collect images of the scene under test, and use the trained YOLO target detection model to detect the collected images to obtain the objects in the scene under test.

[0042] Step 4: Update scene information If the output of step 2 is that the current frame scene is a room, obtain the scene probability corresponding to each object in the current frame in turn , scenario probability The expression is: Special styles in, and is the scene probability corresponding to the previous object; is the attenuation factor; is the number of consecutive discoveries of the same object; is the association probability between the nth object and the i-th scene; is the association probability between the nth object and the jth scene; M is the number of scenes; ; ; N is the number of objects detected in the current frame.

[0043] The scene probability corresponding to the last object in the current frame As the final scene probability of the i-th scene in the current frame ; Use the final scene probability of the previous frame as the scene probability of the current frame .

[0044] The purpose of adding the attenuation factor is to prevent the misidentification of the scene due to continuous observation of a certain object. When the observation list is [cup, cup, cup, cup, cup, cup, computer, computer, tape measure, student power supply], different attenuation factors and the probability of laboratory and bathroom scenes are as follows Figure 3 As shown. If the final scene probability of the current frame Greater than the scene probability threshold When the scene type of the current frame is output as greater than the threshold The final scenario probability The corresponding scene type; if the final scene probability of the current frame Less than the scene probability threshold , re-perform scene recognition.

[0045] If the output of step 2 is that the current frame scene is a corridor, the final scene probability of the current frame Reset to .

[0046] Step 5: Repeat steps 2 to 4 to obtain the scene type corresponding to each frame of image and complete scene recognition.

[0047] Step 6: Use the present invention and the pure visual scene recognition method to perform scene recognition respectively. The resource consumption for medium and light-weight scene recognition is shown in Table 2.

[0048] Table 2 Comparison of resource consumption between the present invention and pure visual scene recognition algorithm It can be seen from Table 2 that compared with the ResNet50 algorithm, the straight line detection module and the scene probability update module of the present invention occupy less space and have significant advantages in reducing resource consumption.

Claims

1. A scene recognition method based on multi-source information fusion, characterized in that: The following steps are involved: Step 1: Use the knowledge graph embedding model to obtain the association probability p between objects and scene types; Step 2: Collect point cloud data of the scene under test; fit the collected point cloud data into multiple straight lines, and preliminarily classify the scene under test according to the number of point clouds in the fitted straight lines; Step 3: Collect images of the scene under test, use the target detection model to detect the collected images, and obtain objects in the scene under test; Step 4: Obtain the final scene probability of the current frame based on the objects in the scene being tested and the corresponding association probability p ; Using the final scene probability Further classify the tested scenarios and obtain the types of the tested scenarios; Step 5: Repeat steps 2 to 4 to obtain the type of the scene being tested in each frame of the image and complete scene recognition.

2. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In the step 2, the preliminary classification method is: count the two fitted straight lines with the largest number of point clouds among multiple fitted straight lines. If the sum of the number of point clouds in these two fitted straight lines exceeds a preset threshold, the measured scene is judged to be a corridor; otherwise, the measured scene is judged to be a room.

3. The scene recognition method based on multi-source information fusion according to claim 2, characterized in that: In step 4, the final scene probability is used The method for further classifying the tested scene is as follows: if the tested scene is determined to be a corridor in step 2, the scene of the current frame is output as a corridor; if the tested scene is determined to be a room in step 2 and the final scene probability of the current frame is Greater than the scene probability threshold When the scene type of the current frame is output as greater than the threshold The final scenario probability The corresponding scene type.

4. The scene recognition method based on multi-source information fusion according to claim 2, characterized in that: In step 4, the final scene probability The method of obtaining is: obtain the scene probability corresponding to each object in the current frame in turn , scenario probability The expression is: in, and is the scene probability corresponding to the previous object; is the attenuation factor; is the number of times the same object is found consecutively; is the association probability between the nth object and the i-th scene; is the association probability between the nth object and the jth scene; M is the number of scenes; ; ; N is the number of objects detected in the current frame; The scene probability corresponding to the last object in the current frame As the final scene probability of the i-th scene in the current frame ; Use the final scene probability of the previous frame as the scene probability of the current frame ; If the scene of the previous frame is a corridor, the probability of the final scene of the previous frame is .

5. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In the step 2, the method of fitting the straight line is as follows: Step 1-1. Convert the collected point cloud data into Cartesian coordinate format; Step 1-2. Fit the point cloud data point by point. If the distance between the extended point and the current fitted line is less than the set range threshold, the extended point is considered to belong to the current fitted line; otherwise, the extended point is used as the starting point of the new fitted line; Step 1-3. Merge two fitted straight lines belonging to the same straight line according to the similarity of slope and intercept.

6. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In the step 1, point cloud data is collected by a single-line laser radar.

7. The scene recognition method based on multi-source information fusion according to claim 6, characterized in that: In the step 2, noise filtering is performed on the point cloud data collected by the single-line laser radar to exclude points in the point cloud data whose distance from the laser radar is less than 0.1 meters or greater than 20 meters.

8. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In the step 1, the method for obtaining the association probability p is: constructing a knowledge graph dataset; Use the knowledge graph dataset to train the knowledge graph embedding model; based on the trained knowledge graph embedding model, obtain the association score x between the object and the scene; and normalize the association score x to obtain the association probability p.

9. The scene recognition method based on multi-source information fusion according to claim 8, characterized in that: In the step 1, during the training process, the expression of the loss function L is set: in, To fix the boundary threshold, control the distance between positive and negative examples; is the defined distance function; is the sigmoid function; h, r, t are the vectors corresponding to the head entity, relation, and tail entity respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector; It is Hadamard.

10. A scene recognition device based on multi-source information fusion, comprising a mounting component and a single-line laser radar, a depth camera and a control module installed on the mounting component; characterized in that: The control module is used to execute a scene recognition method based on multi-source information fusion as described in claim 1; the control module includes a straight line detection module and a scene probability update module; the straight line detection module is used to preliminarily classify the type of the measured scene according to the point cloud data; the scene probability update module is used to further classify the measured scene according to the image of the object.

Images