A scene recognition method and device based on multi-source information fusion

Abstract

This invention discloses a scene recognition method and apparatus based on multi-source information fusion. It performs preliminary classification of the tested scene using data collected by a single-line LiDAR and further classifies the scene using data collected by a depth camera. By combining the detection results of different sensors, it can supplement the information gaps between them. When vision is affected by lighting or there are few visual features in the environment, other sensors can remain stable, ensuring the robustness of the overall recognition. Simultaneously, this invention uses an object detection model to detect objects in the tested scene and combines the associative reasoning capabilities of a knowledge graph to infer the semantics of the detected objects, thereby inferring the scene in which the robot is located. This provides a basis for subsequent task decisions, enabling the robot to select the optimal task execution strategy based on the scene. By comprehensively utilizing multi-dimensional information, the scene recognition accuracy is high.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception and knowledge reasoning technology, specifically relating to a scene recognition method and device based on multi-source information fusion. Background Technology

[0002] With the development of robotics technology, autonomous mobile robots have been widely used in scenarios such as home services, logistics handling, security patrols, and emergency rescue. In these scenarios, robots need to be able to perceive their surrounding environment in real time and adjust their task execution strategies according to environmental changes, which places high demands on the robot's scene recognition and understanding capabilities. For example, a robotic vacuum cleaner will proactively avoid cleaning during human rest hours in the bedroom; it will employ a more powerful cleaning strategy in the kitchen where stubborn stains are prone to accumulate; furthermore, when performing a target search task, if the target is kitchenware, it will prioritize the kitchen and temporarily skip areas with low probability of occurrence, such as the living room. Therefore, scene recognition plays a crucial role in optimizing robot task execution, significantly improving the robot's intelligence level and task execution efficiency.

[0003] Currently, most existing scene recognition methods rely on vision, LiDAR, or other sensors for environmental perception. However, in complex environments (such as hallways and rooms in a home), data from a single sensor is easily affected by external interference, leading to inaccurate or unstable scene classification results. For example, LiDAR may misidentify scenes with similar room structures due to similar features; visual perception may fail due to insufficient lighting or obstruction, or due to low image quality during robot movement, resulting in insufficient image feature information and thus recognition errors. Summary of the Invention

[0004] The purpose of this invention is to provide a scene recognition method based on multi-source information fusion, which comprehensively analyzes the scene through multi-modal information (including LiDAR point cloud data, target detection results, and knowledge reasoning), thereby realizing intelligent understanding and judgment of the robot's environment.

[0005] In a first aspect, the present invention provides a scene recognition method based on multi-source information fusion, which includes the following steps:

[0006] Step 1: Use a knowledge graph embedding model to obtain the association probability p between objects and scenes;

[0007] Step 2: Collect point cloud data of the scene under test; fit the collected point cloud data into multiple straight lines, and perform preliminary classification of the scene under test based on the number of points in the fitted lines;

[0008] Step 3: Acquire images of the scene under test, and use an object detection model to detect objects in the acquired images to obtain the objects in the scene under test;

[0009] Step 4: Obtain the final scene probability of the current frame based on the objects in the tested scene and their corresponding association probabilities p. ; Utilizing the probability of the final scenario Further classify the tested scenarios to obtain their types;

[0010] Step 5: Repeat steps 2 to 4 to obtain the type of the scene being tested in each frame of the image, and complete the scene recognition.

[0011] As a preferred option, in step two, the preliminary classification method is as follows: count the two fitted lines with the most point cloud data among multiple fitted lines. If the sum of the point cloud data in these two fitted lines exceeds a preset threshold, the tested scene is determined to be a corridor; otherwise, the tested scene is determined to be a room.

[0012] Preferably, in step four, the final scene probability is used. The method for further classifying the tested scene is as follows: if step two determines that the tested scene is a corridor, output the scene of the current frame as a corridor; if step two determines that the tested scene is a room and the final scene probability of the current frame is... Greater than the scene probability threshold When the scene type of the current frame is greater than the threshold, output the scene type of the current frame. Final scenario probability The corresponding scene type.

[0013] Preferably, in step four, the final scene probability The method for obtaining the probability is as follows: sequentially obtain the scene probability corresponding to each object in the current frame. Scenario probability The expression is:

[0014]

[0015] in, and This represents the scene probability corresponding to the previous object; It is the attenuation factor; The number of times the same object is found consecutively; Let be the probability of association between the nth object and the ith scene. Let M be the probability of association between the nth object and the jth scene; M is the number of scenes. ; N is the number of objects detected in the current frame;

[0016] The scene probability corresponding to the last object in the current frame The final scene probability of the i-th scene in the current frame The final scene probability of the previous frame is used as the scene probability of the current frame. If the previous frame's scene was a corridor, the probability of the previous frame's final scene being... .

[0017] Preferably, in step two, the method for fitting the straight line is as follows:

[0018] Step 1-1. Convert the collected point cloud data into Cartesian coordinate format;

[0019] Step 1-2. Fit the point cloud data point by point. If the distance between the extended point and the current fitted line is less than the set range threshold, the extended point is considered to belong to the current fitted line; otherwise, the extended point is used as the starting point of the new fitted line.

[0020] Steps 1-3. Merge two fitted lines that belong to the same line based on slope similarity and intercept similarity.

[0021] Preferably, in step one, the point cloud data is collected using a single-line lidar.

[0022] Preferably, in step two, noise filtering is performed on the point cloud data collected by the single-line lidar to exclude points in the point cloud data that are less than 0.1 meters or more than 20 meters away from the lidar.

[0023] Preferably, in step one, the method for obtaining the association probability p is as follows: constructing a knowledge graph dataset; training a knowledge graph embedding model using the knowledge graph dataset; obtaining the association score x between objects and scenes based on the trained knowledge graph embedding model; and obtaining the association probability p after normalizing the association score x.

[0024] Preferably, in step one, the expression for the loss function L during training is set as follows:

[0025]

[0026]

[0027] in, To control the distance between positive and negative examples by setting a fixed boundary threshold; The distance function is defined. is the sigmoid function; h, r, and t are the vectors corresponding to the head entity, relation, and tail entity, respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set, respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector. For Hadama accumulation.

[0028] Secondly, the present invention provides a scene recognition device based on multi-source information fusion, which includes a mounting component and a single-line lidar, a depth camera, and a control module mounted on the mounting component; the control module is used to execute the above-mentioned scene recognition method; the control module includes a line detection module and a scene probability update module; the line detection module is used to perform preliminary classification of the type of the tested scene based on point cloud data; the scene probability update module is used to further classify the tested scene based on the image of the object.

[0029] The beneficial effects of this invention are:

[0030] 1. This invention uses data collected by a single-line lidar to initially classify the scene under test, and then uses data collected by a depth camera to further classify the scene. By combining the detection results of different sensors, it can supplement the information gaps between different sensors. When vision is affected by light or there are few visual features in the environment, other sensors can remain stable, ensuring the robustness of the overall recognition.

[0031] 2. This invention uses an object detection model to detect objects in the tested scene and combines the association reasoning ability of a knowledge graph to infer the semantics of the detected objects, thereby inferring the scene in which the robot is located. This provides a basis for subsequent task decision-making, enabling the robot to select the optimal task execution strategy according to the scene. By comprehensively utilizing multi-dimensional information, the scene recognition accuracy is high.

[0032] 3. This invention can directly use the output of the target detection algorithm to perform scene inference without having to perform multiple inferences on the original image using deep learning, which greatly reduces the consumption of computing resources and time, and is suitable for scenarios with limited computing resources. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the scene recognition device in this invention.

[0035] Figure 2 This is an overall flowchart of the scene recognition method in this invention.

[0036] Figure 3 The diagrams illustrate the probability update process for different scenarios; (a) illustrates the probability update process for the laboratory scenario; and (b) illustrates the probability update process for the toilet scenario. Detailed Implementation

[0037] The technical solutions provided by the present invention will be described in detail below with reference to specific examples. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0038] like Figure 1 As shown, a scene recognition device based on multi-source information fusion includes a mounting component and a single-line LiDAR, a depth camera, and a control module mounted on the mounting component. The single-line LiDAR is used to collect point cloud data of the scene under test. The depth camera is used to collect images of objects in the scene under test. The control module includes a line detection module and a scene probability update module. The line detection module is used to perform preliminary classification of the type of the scene under test based on the point cloud data. The scene probability update module is used to further classify the type of the scene under test based on the images of the objects.

[0039] like Figure 2 As shown, a scene recognition method based on multi-source information fusion, using the aforementioned scene recognition device, includes the following steps:

[0040] Step 1: Obtain the association probability between objects in the scene and the scene itself.

[0041] 1-1. Construct a knowledge graph dataset, where relation attributes include object instances, object categories, and room types. Relationship attributes must include at least two of the following: the relationship between an object instance and its category (e.g., "belongs to"), and the relationship between an object instance and its room (e.g., "is in the room"). Furthermore, some unknown entities are allowed, which can be automatically filled in by the model during subsequent reasoning. A partial knowledge graph dataset is shown in Table 1 in this example.

[0042] Table 1 Knowledge Graph Dataset

[0043]

[0044] 1-2. The knowledge graph is embedded using the RotateE model, and self-adversarial negative sampling techniques are employed to improve the model's efficiency and effectiveness. The goal is to minimize the triple score function, resulting in higher scores for true triples and lower scores for false triples. The expression for the loss function L is:

[0045] (1)

[0046] (2)

[0047] in, To control the distance between positive and negative examples by setting a fixed boundary threshold; The distance function is defined. is the sigmoid function used to map the scores to the range [0,1]; h, r, and t are the vectors corresponding to the head entity, relation, and tail entity, respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set, respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector; For Hadama accumulation.

[0048] 1-3. Based on the trained RotateE model, perform tail inference tasks to obtain the association scores x between various objects and the scene.

[0049] 1-4. Normalize the association score x using formula (9) to construct a database of association probabilities between objects and scenes.

[0050] (3)

[0051] in, This represents the probability of association between the object and the scene.

[0052] Step 2: Preliminary classification of the types of the tested scenarios

[0053] 2-1. Use a single-line lidar to collect point cloud data of the scene under test; perform noise filtering on the lidar scanning data to exclude points in the point cloud that are less than 0.1 meters or more than 20 meters away from the lidar, so as to reduce the interference of measurement noise on subsequent processing and ensure the validity of the point cloud data.

[0054] 2-2. Convert the polar coordinate data (including the angle range of the laser beam and ranging data) collected by the lidar into Cartesian coordinate format (x, y) of a two-dimensional point cloud. Then, based on the distance r between each point and the lidar and the corresponding angle... The position information in the Cartesian coordinate system is calculated as shown in formula (1):

[0055] Special Style (4)

[0056] 2-3. Utilizing the ordered nature of the point cloud output by LiDAR scanning, all straight line segments are identified using an incremental straight line fitting method: starting from the starting point of the point cloud data, fitting is performed point by point, and the least squares method is used to fit the lines to detect continuous straight line segments. The straight line point cloud cluster is... ;where x i and y i ... Representing the point cloud set in matrix form:

[0057] (5)

[0058] The expression for the parameters of the line fitted using the least squares algorithm is:

[0059] (6)

[0060] Where m is the slope; c is the intercept; and T is the transpose sign.

[0061] The point currently added to the current fitted line is taken as the extension point. If the extension point satisfies formula (4), that is, the distance between the extension point and the current fitted line is less than the set range threshold, then the extension point is considered to belong to the current fitted line; otherwise, the extension point is taken as the starting point of the new fitted line.

[0062] (7)

[0063] in, The coordinates of the extended point; The set range threshold.

[0064] 2-4. Merge fitted lines that belong to the same line based on slope similarity and intercept similarity; if two fitted lines satisfy both formulas (5) and (6), then these two fitted lines are considered to belong to the same line.

[0065] (8)

[0066] (9)

[0067] Where m1 and m2 are the slopes of the two lines, respectively; c1 and c2 are the intercepts of the two lines, respectively. and These are the slope similarity threshold and the intercept similarity threshold, respectively.

[0068] 2-5. Statistically select the two fitted lines that contain the most point cloud data. If the sum of the point cloud data in these two fitted lines exceeds a preset threshold, the tested scene is determined to be a corridor; otherwise, the tested scene is determined to be a room.

[0069] In this embodiment, the horizontal field of view of the single-line lidar is 270°, the scanning frequency is 30Hz, the number of point clouds in a single scan cycle is 1440, the detection distance is 25m, the corridor width is 3.72m, and the point cloud number threshold is set to 300.

[0070] Step 3: Detect objects in the scene being tested

[0071] 3-1. Construct a target image dataset, which includes images of various objects and their annotation information. Divide the target image dataset into a training set, a validation set, and a test set in a ratio of 7:2:1. When dividing the dataset, ensure that images of different scenes and categories are evenly distributed in the training set and the validation set to avoid bias in model training towards a certain type of data.

[0072] 3-2. Train the YOLO object detection model using the training set; set the training epochs to 100 to ensure the model fully learns the features of different scenarios; use a learning rate scheduler to dynamically adjust the learning rate during training to avoid overfitting or underfitting. Incorporate an early stopping mechanism to dynamically terminate training based on the accuracy of the validation set, preventing overtraining from causing a performance degradation issue.

[0073] 3-3. Use a depth camera to acquire images of the scene under test, and use the trained YOLO object detection model to detect objects in the acquired images to obtain the objects in the scene under test.

[0074] Step 4: Update scene information

[0075] If step two outputs that the current frame scene is a room, then the scene probability corresponding to each object in the current frame is obtained sequentially. Scenario probability The expression is:

[0076] Special styles

[0077] in, and This represents the scene probability corresponding to the previous object; It is the attenuation factor; The number of times the same object is found consecutively; Let be the probability of association between the nth object and the ith scene. Let M be the probability of association between the nth object and the jth scene; M is the number of scenes. ; N represents the number of objects detected in the current frame.

[0078] The scene probability corresponding to the last object in the current frame The final scene probability of the i-th scene in the current frame The final scene probability of the previous frame is used as the scene probability of the current frame. .

[0079] The purpose of adding an attenuation factor is to prevent misidentification of a scene due to continuous observation of a single object. When the observation list is [cup, cup, cup, cup, cup, cup, computer, computer, measuring tape, student power supply], the probabilities of different attenuation factors and the laboratory and restroom scenes are as follows: Figure 3 As shown. If the final scene probability of the current frame... Greater than the scene probability threshold When the scene type of the current frame is greater than the threshold, output the scene type of the current frame. Final scenario probability The corresponding scene type; if the final scene probability of the current frame Less than the scene probability threshold Then, scene recognition is performed again.

[0080] If step two outputs that the current frame scene is a corridor, then the final scene probability of the current frame will be... Reset to .

[0081] Step 5: Repeat steps 2 to 4 to obtain the scene type corresponding to each frame of the image and complete the scene recognition.

[0082] Step 6: Use the present invention and the pure visual scene recognition method to perform scene recognition respectively. The resource consumption for medium-lightweight scene recognition is shown in Table 2.

[0083] Table 2. Resource consumption comparison between the present invention and pure vision scene recognition algorithms.

[0084]

[0085] As can be seen from Table 2, compared with the ResNet50 algorithm, the line detection module and scene probability update module of the present invention occupy less space and have a significant advantage in reducing resource consumption.

Claims

1. A scene recognition method based on multi-source information fusion, characterized in that: Includes the following steps: Step 1: Use a knowledge graph embedding model to obtain the association probability p between objects and scene types; Step 2: Collect point cloud data of the scene under test; fit the collected point cloud data into multiple straight lines, and perform preliminary classification of the scene under test based on the number of points in the fitted lines; In step two, the preliminary classification method is as follows: count the two fitted lines with the most point cloud data among multiple fitted lines. If the sum of the point cloud data in these two fitted lines exceeds a preset threshold, the tested scene is determined to be a corridor; otherwise, the tested scene is determined to be a room. Step 3: Acquire images of the scene under test, and use an object detection model to detect objects in the acquired images to obtain the objects in the scene under test; Step 4: Obtain the final scene probability of the current frame based on the objects in the tested scene and their corresponding association probabilities p. ; Using the final scenario probability Further classify the tested scenarios to obtain their types; In step four, the final scene probability The method for obtaining the probability is as follows: sequentially obtain the scene probability corresponding to each object in the current frame. Scenario probability The expression is: ； in, and This represents the scene probability corresponding to the previous object; It is the attenuation factor; The number of times the same object is found consecutively; Let be the probability of association between the nth object and the ith scene. Let M be the probability of association between the nth object and the jth scene; M is the number of scenes. ; N is the number of objects detected in the current frame; The scene probability corresponding to the last object in the current frame The final scene probability of the i-th scene in the current frame The final scene probability of the previous frame is used as the scene probability of the current frame. If the previous frame's scene was a corridor, the probability of the previous frame's final scene being... ; Step 5: Repeat steps 2 to 4 to obtain the type of the scene being tested in each frame of the image, and complete the scene recognition.

2. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In step four, the final scene probability is used. The method for further classifying the tested scene is as follows: if step two determines that the tested scene is a corridor, output the scene of the current frame as a corridor; if step two determines that the tested scene is a room and the final scene probability of the current frame is... Greater than the scene probability threshold When the scene type of the current frame is greater than the threshold, output the scene type of the current frame. Final scenario probability The corresponding scene type.

3. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In step two, the method for fitting a straight line is as follows: Step 1-1. Convert the collected point cloud data into Cartesian coordinate format; Step 1-2. Fit the point cloud data point by point. If the distance between the extended point and the current fitted line is less than the set range threshold, the extended point is considered to belong to the current fitted line; otherwise, the extended point is used as the starting point of the new fitted line. Steps 1-3. Merge two fitted lines that belong to the same line based on slope similarity and intercept similarity.

4. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In step one, point cloud data is collected using a single-line lidar.

5. The scene recognition method based on multi-source information fusion according to claim 4, characterized in that: In step two, noise filtering is performed on the point cloud data collected by the single-line lidar to exclude points in the point cloud data that are less than 0.1 meters or more than 20 meters away from the lidar.

6. The scene recognition method based on multi-source information fusion according to claim 1, characterized in that: In step one, the method for obtaining the association probability p is: constructing a knowledge graph dataset; The knowledge graph embedding model is trained using a knowledge graph dataset; based on the trained knowledge graph embedding model, the association score x between objects and scenes is obtained; the association probability p is obtained after normalizing the association score x.

7. The scene recognition method based on multi-source information fusion according to claim 6, characterized in that: In step one, during the training process, the expression for the loss function L is set as follows: ； ； in, To control the distance between positive and negative examples by setting a fixed boundary threshold; The distance function is defined. is the sigmoid function; h, r, and t are the vectors corresponding to the head entity, relation, and tail entity, respectively; h' and t' are the vectors corresponding to the head entity and tail entity in the negative sample set, respectively; N is the negative sample set obtained by negative sampling; d is the dimension of the vector. For Hadama accumulation.

8. A scene recognition device based on multi-source information fusion, comprising a mounting component and a single-line lidar, a depth camera, and a control module mounted on the mounting component; characterized in that: The control module is used to execute the scene recognition method based on multi-source information fusion as described in claim 1; the control module includes a line detection module and a scene probability update module; the line detection module is used to perform preliminary classification of the type of the scene under test based on point cloud data; the scene probability update module is used to further classify the scene under test based on the image of the object.

Images