Indoor article rapid identification method based on scene understanding

Abstract

The invention belongs to the technical field of visual recognition and real-time detection of indoor robots, and particularly discloses an indoor article rapid recognition method based on scene understanding, which comprises the following steps: firstly, constructing a scene understanding module and a dynamic judgment module, and judging the type of a current indoor scene; secondly, based on a scene classification result, dynamically judging and calling a pre-training recognition model matched with the scene; dynamically optimizing an identification strategy of the article identification network; and finally, calling the most matched intelligent identification network to detect and classify the indoor articles. According to the indoor article rapid identification method based on scene understanding, through continuous iterative optimization of the model in the article detection process, gradient information and a Laplace enhancement technology are combined, so that the model can adjust own parameters in real time to adapt to identification requirements in different scenes, and the identification efficiency is improved. The flexibility not only improves the overall performance of the model, but also provides a feasible solution for wide application of indoor article identification.

Description

Technical Field

[0001] The present invention relates to the technical field of visual recognition and real-time detection of indoor robots, and in particular to a method for rapid recognition of indoor objects based on scene understanding. Background Art

[0002] Current indoor object recognition technology mainly relies on two types of methods: RFID-based systems and models trained based on large data sets. On the one hand, the limitation of RFID-based systems is that each item needs to be labeled and cannot adapt to new environments, which limits the possibility of its widespread application and leads to limitations and inefficiencies in object recognition. In addition, such systems are difficult to effectively cope with the diversity and changes of items in actual deployment, resulting in low recognition rates. On the other hand, although deep learning models trained with large-scale data sets based on deep learning have achieved certain success in some scenarios, they still face multiple challenges. First, these models are usually based on general architectures and are difficult to optimize for specific scenarios, resulting in unsatisfactory performance in complex environments. Second, training such large models requires a lot of computing resources and time, which makes them face many challenges in practical applications, which is particularly unfavorable for resource-constrained devices. Summary of the invention

[0003] The purpose of the present invention is to provide a method for rapid indoor object recognition based on scene understanding to solve the challenges encountered by existing indoor object recognition technologies when dealing with complex environments, including improving the model's adaptability to environmental changes to achieve accurate recognition of diverse objects; reducing dependence on large amounts of labeled data to enhance the model's application potential in unknown indoor environments; expanding the generalization range of the model so that it can better cope with object recognition tasks in different indoor scenes; and introducing effective mechanisms to optimize the ability to handle the dynamics of objects in indoor environments and complex backgrounds.

[0004] To achieve the above object, the present invention provides a method for quickly identifying indoor objects based on scene understanding, comprising the following steps:

[0005] S1. Build a scene understanding module and a dynamic judgment module to determine the type of the current indoor scene;

[0006] S2. Based on the scene classification results, dynamically determine and call the pre-trained recognition model that matches the scene;

[0007] S3, dynamically optimize the recognition strategy of the object recognition network;

[0008] S4: Call the most matching intelligent recognition network to detect and classify indoor objects.

[0009] Preferably, in S1, a scene understanding module based on a lightweight network MobileNetV3 is constructed to simulate the cognitive mechanism of humans when dealing with complex indoor scenes; the scene understanding module uses visual information to infer the preprocessed image, judge the type of the current indoor scene, and output the scene classification result, wherein the scene types include bedroom, kitchen or bathroom.

[0010] Preferably, in S1, the cognitive mechanism of humans in dealing with complex indoor scenes includes multi-level processing mechanism, memory and reasoning mechanism, and feedback adjustment mechanism;

[0011] The memory and reasoning mechanism is used to capture scene features and make up for missing visual information, infer the shape and position of objects, and the feedback adjustment mechanism is used to optimize the memory and reasoning mechanism.

[0012] Preferably, a deep network model based on memory and reasoning mechanism is constructed, specifically:

[0013] The feature input is obtained through 1x1 convolution, and then local information is extracted through 3x3 deep convolution. The extracted local features are summarized through the pooling layer to form the input of the memory and reasoning mechanism. The global features of the scene are gradually strengthened through layer-by-layer memory update and reasoning operations. The enhanced global features are processed through the fully connected layer and finally output through the last fully connected layer to generate the final features.

[0014] Preferably, in S2, the pre-trained recognition model adopts an improved YOLOv7 model, and the process of improving the YOLOv7 model is specifically as follows:

[0015] The CBAM module is introduced into the backbone and head parts of the YOLOv7 model. The CBAM module includes the channel attention module CAM and the spatial attention module SAM;

[0016] Among them, the spatial attention module SAM is used for the model to focus on a specific position of the image and extract more distinctive features. The channel attention module CAM is used to automatically learn and focus on the most important feature channels to perform object detection and recognition in different scenarios.

[0017] Preferably, in S3, a feedback adjustment mechanism is used to dynamically optimize the identification strategy of the object identification network, specifically:

[0018] During the object recognition process, the object recognition strategy is dynamically adjusted through the feedback adjustment mechanism, and the model is continuously iterated and optimized in combination with gradient information during the object recognition process. By analyzing gradient information and visual feedback, the parameters and weights of the detection model are automatically adjusted, and combined with Laplace enhancement technology, the edges and details of objects are captured.

[0019] Preferably, in S4, after the scene understanding and dynamic judgment module completes the preliminary judgment, the most matching recognition network is called, and the recognition network will detect and classify indoor objects based on the information obtained in the scene understanding module, and verify and optimize the recognition results through evaluation indicators.

[0020] Preferably, CIOU is used as the evaluation indicator:

[0021]

[0022] Among them, ρ 2 (b,b gt ) represents the Euclidean distance between the center point of the predicted box and the center point of the real box, c represents the diagonal distance of the minimum closed area that contains both the predicted box and the real box, α represents the weight coefficient, and v quantifies the difference in aspect ratio between the predicted box and the real box by comparing the aspect ratio of the predicted box and the real box;

[0023] The calculation formulas for α and v are as follows:

[0024]

[0025] Among them, w and h represent the width and height of the prediction box respectively, w gt and h gt Represent the width and height of the real box respectively.

[0026] Therefore, the present invention adopts the above-mentioned method for rapid identification of indoor objects based on scene understanding, and the beneficial effects are as follows:

[0027] (1) The present invention achieves efficient and accurate object recognition by introducing the algorithm framework of MobileNetV3 and improving YOLOv7.

[0028] (2) The present invention solves the problems of insufficient recognition accuracy, slow speed and strong dependence on large-scale data sets in the prior art in complex environments. Through the scene classification of MobileNetV3 and the improved precise detection of YOLOv7, it can effectively identify objects in different scenes such as bedrooms, bathrooms and kitchens. Compared with the prior art, the accuracy is improved by 1.2%, 5.6% and 4.6%, and a high frame rate is maintained, which is suitable for real-time detection.

[0029] (3) The present invention significantly enhances the robustness in complex scenes and reduces false detection and missed detection by introducing the CBAM attention mechanism. At the same time, the combination of the feedback adjustment mechanism and gradient information enables the model to be dynamically optimized, improves the recognition ability of small objects and occluders, reduces the dependence on large-scale data sets, and significantly improves the generalization ability.

[0030] (4) The method provided by the present invention does not require complex model tuning, saves development and testing time, and improves economic benefits in practical applications.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an overall flow chart of an embodiment of a method for rapid identification of indoor objects based on scene understanding of the present invention;

[0033] Figure 2 It is a structural block diagram of a cognitive mechanism of human beings in dealing with a complex indoor environment according to an embodiment of a method for rapid identification of indoor objects based on scene understanding of the present invention;

[0034] Figure 3 It is an overall algorithm structure diagram of an embodiment of a method for rapid indoor object recognition based on scene understanding of the present invention;

[0035] Figure 4 It is a schematic diagram of the overall model structure after the improved YOLOv7 model and the addition of the CMAM module according to an embodiment of a method for rapid indoor object recognition based on scene understanding of the present invention;

[0036] Figure 5 It is a block diagram of a deep network structure based on a memory reasoning mechanism in an embodiment of a method for rapid indoor object recognition based on scene understanding of the present invention;

[0037] Figure 6 It is a schematic diagram of the CBMA module structure of an embodiment of a method for rapid indoor object recognition based on scene understanding of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0039] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0040] like Figure 1 As shown, a method for rapid indoor object recognition based on scene understanding includes the following steps:

[0041] S1. Build a scene understanding module and a dynamic judgment module to determine the type of the current indoor scene;

[0042] Data collection: Use cameras and other devices to collect pictures or videos in different environments, and record data on indoor objects to form a training set.

[0043] The types of items in the training set depend on the annotations of the dataset. The more types of annotated items and the larger the number of training images, the better the effect. However, too large a dataset requires high server computing power. Therefore, the balance between model effect and computing power requirements needs to be determined through experiments to achieve the best results.

[0044] Feature extraction: Use the lightweight network MobileNetV3 to extract low-dimensional features of the image for subsequent classification and recognition. In order to solve the problem of complex categories and diverse layouts of objects in indoor environments, a scene understanding module based on the lightweight network MobileNetV3 is constructed. This module is used to simulate the cognitive mechanism of humans when dealing with complex indoor scenes, that is, the initial perception process; the scene understanding module uses visual information to infer the preprocessed image, determine the type of the current indoor scene, and output the scene classification results, where the scene types include bedroom, kitchen or bathroom, etc. Through the rapid classification of the scene, the optimal model is automatically selected, which enhances the system's ability to cope with complex environments.

[0045] This step is the basis of the entire recognition process. Scene classification determines the selection and processing logic of subsequent recognition models. Since object recognition in different scenes requires different model processing, scene classification can significantly improve the accuracy and efficiency of object recognition.

[0046] This process can significantly reduce the reliance on large-scale data sets and improve recognition efficiency. At this stage, the output of scene understanding will serve as the basis for subsequent object recognition and select the optimal recognition model for different scenarios.

[0047] like Figure 2 As shown in the figure, the cognitive mechanism of humans in dealing with complex indoor scenes includes multi-level processing mechanism, memory and reasoning mechanism and feedback adjustment mechanism. These three mechanisms work together to help the brain recognize and understand different scenes and objects. The memory and reasoning mechanism is used to capture scene features and make up for missing visual information, so that the system can infer the shape and position of objects, thereby improving the efficiency and accuracy of subsequent scene recognition. The feedback adjustment mechanism is used to further optimize the memory and reasoning mechanism to ensure the accuracy and robustness of judgment, demonstrating the powerful ability of the brain to cope with complex environments.

[0048] like Figure 5 As shown in the figure, a deep network model based on memory and reasoning mechanism is constructed, specifically:

[0049] The feature input is obtained through 1x1 convolution (1x1, NL), and then the local information is extracted through 3x3 depth convolution (3x3 Dwis, NL). The extracted local features are aggregated through the pooling layer to form the input of the memory and reasoning mechanism. Through layer-by-layer memory update and reasoning operations, the global features of the scene are gradually strengthened. The enhanced global features are then processed through a fully connected layer (FC, ReLU) and finally output through a fully connected layer (FC, hard-σ) to generate the final features.

[0050] S2. Build an improved YOLOv7 model:

[0051] Based on the MobileNetV3 extraction, the improved YOLOv7 is used for training to identify the categories of objects in the picture; that is, based on the scene classification results, the system will dynamically judge and call the pre-trained recognition model that matches the scene. Specifically, the pre-trained recognition model uses the improved YOLOv7 model to perform accurate object detection and recognition in different scenes. The process of improving the YOLOv7 model is as follows:

[0052] The traditional YOLOv7 model consists of three parts: input, Backbone network and Head. After preprocessing the input part, the image enters the Backbone network to extract features, and obtains features of different sizes through feature fusion processing, and finally sends it to the detection head and outputs the recognition result.

[0053] The improved YOLOv7 model introduces the CBAM module into the backbone and head parts of the traditional YOLOv7 model, such as Figure 6 As shown in the figure, the CBAM module includes a channel attention module CAM and a spatial attention module SAM, which combines the channel and spatial attention mechanisms to improve the attention to key item features.

[0054] This mechanism plays a vital role in improving the accuracy of object recognition in complex environments, especially when there is a lot of background noise in the scene. The CBAM module can help the model ignore irrelevant parts and focus on useful information, ensuring that the model can handle complex object layouts.

[0055] Among them, the spatial attention module SAM is used for the model to focus on specific positions of the image and extract more distinguishing features, while the channel attention module CAM is used to automatically learn and focus on the most important feature channels, thereby improving the accuracy of object detection and recognition in different scenarios. The overall CBAM module enhances the model's ability to represent key features of objects, especially in complex indoor environments. By focusing on channels and spatial positions, the model can identify objects more accurately.

[0056] Therefore, the improved YOLOv7 model automatically focuses on key channels and spatial areas through fine-grained feature extraction combined with the CBAM attention mechanism, thereby increasing attention to details and diverse objects; through this dynamic scheduling method, the computational burden of the unified model is reduced, and the adaptability and generalization ability of the model in different scenarios are improved.

[0057] S3, model training: Use the collected data set to train the MobileNetV3-YOLOv7 joint model, optimize the network weights, and improve the recognition effect.

[0058] Specifically, in order to further improve the recognition accuracy of the model, the present invention adopts a feedback adjustment mechanism to dynamically optimize the recognition strategy of the object recognition network, specifically:

[0059] In the object recognition process, the object recognition strategy is dynamically adjusted through the feedback adjustment mechanism. The model is continuously optimized iteratively in the object recognition process by combining gradient information. By analyzing gradient information and visual feedback, the system continuously and automatically adjusts the parameters and weights of the detection model to gradually adapt it to the object recognition task in the current scene. In addition, the Laplace enhancement technology is combined to improve the ability to capture the edges and details of objects. The model's dependence on large-scale data sets is significantly reduced, and the generalization ability is improved, thereby improving the overall detection accuracy and reducing the recognition bias caused by environmental changes.

[0060] like Figure 3 As shown in the figure, in the lightweight MobileNetV3 architecture, the model can better understand the global information in the scene through layer-by-layer feature extraction and accumulation, combined with memory and reasoning mechanisms. This mechanism aggregates features at different levels and updates them through memory units to ensure that important information in the scene is retained, thereby enhancing the model's ability to understand complex scenes. In the improved YOLOv7 object recognition module, the feedback adjustment mechanism combines the loss function and additional regularization terms, and the Laplace operator dynamically adjusts the accuracy of object recognition.

[0061] like Figure 4 As shown in the figure, the improved YOLOv7 model optimizes the target detection performance by introducing gradient feedback mechanism and Laplace operator to strengthen image features and carrying CBAM module to enhance key feature recognition.

[0062] S4, object recognition prediction: Input the image to be recognized, extract features through the lightweight network MobileNetV3, and use the improved YOLOv7 model to detect and classify indoor objects and output the object category;

[0063] Specifically, when the scene understanding and dynamic judgment module completes the preliminary judgment, the system will call the most matching recognition network. The recognition network will quickly and accurately detect and classify indoor objects based on the information obtained from the scene understanding module.

[0064] Model optimization: Based on the recognition effect and feedback adjustment mechanism, data is continuously collected to fine-tune the model and improve recognition accuracy. The recognition results are verified and optimized through the evaluation index system to ensure that the final output results are highly accurate and robust.

[0065] First, use IOU (Intersection Over Union) as the evaluation indicator, as shown in formula (1):

[0066]

[0067] Among them, B P represents the area of ​​the prediction box, B gt Represents the area of ​​the true box, so the numerator of IOU is the union of the predicted box and the true box, and the denominator is the intersection of the predicted box and the true box. However, there is an obvious problem with IOU, that is, when the two boxes have no intersection, there is no way to calculate the gradient.

[0068] Therefore, the present invention uses CIOU (Complete Intersection over Union) as the evaluation index, as shown in formula (2), which not only solves the problem of back propagation failure when the predicted box and the real box do not overlap when IOU, but also solves the problem that GIOU (Generalized Intersection over Union) does not consider the aspect ratio, so it is currently a better evaluation method:

[0069]

[0070] Among them, ρ 2 (b,b gt ) represents the Euclidean distance between the center point of the predicted box and the center point of the real box, c represents the diagonal distance of the minimum closed area that contains both the predicted box and the real box, α represents the weight coefficient, and v quantifies the difference in aspect ratio between the predicted box and the real box by comparing the aspect ratio of the predicted box and the real box;

[0071] The calculation formulas for α and v are as follows:

[0072]

[0073] Among them, w and h represent the width and height of the prediction box respectively, w gt and h gtRepresent the width and height of the real box respectively. Calculate the overlap between the predicted box and the real box. In order to deal with the problem that IOU cannot be back-propagated when the predicted box and the real box have no intersection, the CIOU (Complete IOU) evaluation method is used. This method considers the center distance and aspect ratio of the predicted box and the real box at the same time to ensure that the model has better performance during training.

[0074] Therefore, the present invention adopts the above-mentioned method for rapid indoor object recognition based on scene understanding, and proposes multiple specific implementation plans to address the challenges of object recognition in complex indoor environments, thereby improving the accuracy, robustness and efficiency of recognition.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for rapid indoor object recognition based on scene understanding, characterized in that: The following steps are involved: S1. Build a scene understanding module and a dynamic judgment module to determine the type of the current indoor scene; S2. Based on the scene classification results, dynamically determine and call the pre-trained recognition model that matches the scene; S3, dynamically optimize the recognition strategy of the object recognition network; S4: Call the most matching intelligent recognition network to detect and classify indoor objects.

2. According to claim 1, a method for rapid identification of indoor objects based on scene understanding is characterized in that: In S1, a scene understanding module based on the lightweight network MobileNetV3 is constructed to simulate the cognitive mechanism of humans when dealing with complex indoor scenes; the scene understanding module uses visual information to infer the preprocessed images, determine the type of the current indoor scene, and output the scene classification results, where the scene types include bedroom, kitchen or bathroom.

3. The method for rapid indoor object recognition based on scene understanding according to claim 2, characterized in that: In S1, the cognitive mechanisms of humans in dealing with complex indoor scenes include multi-level processing mechanisms, memory and reasoning mechanisms, and feedback adjustment mechanisms; The memory and reasoning mechanism is used to capture scene features and make up for missing visual information, infer the shape and position of objects, and the feedback adjustment mechanism is used to optimize the memory and reasoning mechanism.

4. The method for rapid indoor object recognition based on scene understanding according to claim 3 is characterized in that: Construct a deep network model based on memory and reasoning mechanism, specifically: The feature input is obtained through 1x1 convolution, and then local information is extracted through 3x3 deep convolution. The extracted local features are summarized through the pooling layer to form the input of the memory and reasoning mechanism. The global features of the scene are gradually strengthened through layer-by-layer memory update and reasoning operations. The enhanced global features are processed through the fully connected layer and finally output through the last fully connected layer to generate the final features.

5. The method for rapid indoor object recognition based on scene understanding according to claim 4, characterized in that: In S2, the pre-trained recognition model uses the improved YOLOv7 model. The process of improving the YOLOv7 model is as follows: The CBAM module is introduced into the backbone and head parts of the YOLOv7 model. The CBAM module includes the channel attention module CAM and the spatial attention module SAM; Among them, the spatial attention module SAM is used for the model to focus on specific locations of the image and extract more discriminative features; The channel attention module CAM is used to automatically learn and focus on the most important feature channels to perform object detection and recognition in different scenarios.

6. The method for rapid indoor object recognition based on scene understanding according to claim 4, characterized in that: In S3, a feedback adjustment mechanism is used to dynamically optimize the recognition strategy of the object recognition network. Specifically: During the object recognition process, the object recognition strategy is dynamically adjusted through the feedback adjustment mechanism, and the model is continuously iterated and optimized in combination with gradient information during the object recognition process. By analyzing gradient information and visual feedback, the parameters and weights of the detection model are automatically adjusted, and combined with Laplace enhancement technology, the edges and details of objects are captured.

7. The method for rapid indoor object recognition based on scene understanding according to claim 6, characterized in that: In S4, after the scene understanding and dynamic judgment module completes the preliminary judgment, the most matching recognition network is called. The recognition network will detect and classify indoor objects based on the information obtained in the scene understanding module, and verify and optimize the recognition results through evaluation indicators.

8. The method for rapid indoor object recognition based on scene understanding according to claim 7, characterized in that: Using CIOU as the evaluation indicator: Among them, ρ 2 (b,b gt ) represents the Euclidean distance between the center point of the predicted box and the center point of the real box, c represents the diagonal distance of the minimum closed area that contains both the predicted box and the real box, α represents the weight coefficient, and v quantifies the difference in aspect ratio between the predicted box and the real box by comparing the aspect ratio of the predicted box and the real box; The calculation formulas for α and v are as follows: Among them, w and h represent the width and height of the prediction box respectively, w gt and h gt Represent the width and height of the real box respectively.

Images