A knowledge graph-based scene recognition method, a terminal device, and a storage medium

Abstract

The application relates to a knowledge graph-based scene recognition method, a terminal device and a storage medium, wherein a light graph detection network based on target relationship knowledge learning is constructed in the method, target recognition can be performed according to target relationship knowledge, and light processing is performed through a model structure reconstruction method; through adding a semantic analysis module, the relationship knowledge or attribute knowledge determined in a triple combination can be used to enhance features on the basis of traditional detection. The application improves recognition accuracy, performs light processing on a model, solves the problem that similar scenes are difficult to distinguish, and is suitable for being deployed on a service robot to perform scene recognition.

Description

Technical Field

[0001] This invention relates to the field of target recognition, and more particularly to a scene recognition method, terminal device, and storage medium based on knowledge graphs. Background Technology

[0002] With the rapid development of artificial intelligence technology, service robots are being widely applied in various industries. In the catering industry, service robots can be used for tasks such as taking orders, delivering food, and cleaning tables; in the medical field, they can be used to transport medicines and medical equipment, monitor patients' health, provide rehabilitation treatment, and offer nursing services; in the retail industry, they can be used to guide customers, provide information and advice, and perform tasks such as inventory management and checkout; in the security field, they can be used for patrolling, monitoring, and alarm functions. There are even cases where a single robot can perform multiple functions. For a service robot to play different roles in different scenarios, it must first accurately identify the current scenario type. Therefore, the scene recognition technology that can be deployed and applied to service robots is an issue that requires attention.

[0003] Scene images contain multiple objects, as well as multi-layered information such as global layout, background environment, spatial position and semantic relationship between objects. Traditional manual scene recognition mainly uses various simple low-level features, but such scene recognition methods have little semantic information and very limited expressive power.

[0004] Research on scene recognition methods has gradually evolved to the stage based on deep learning. Considering the increased complexity of scene recognition, many researchers have modified network structures to address the shortcomings of classic network architectures in scene recognition, proposing new learning methods and network structures. These new structures can automatically extract more semantic and structural features from scene images, extracting both global and local features, fusing and classifying these features to compensate for missing scene image information and improve the accuracy of scene recognition. However, problems such as low recognition accuracy or excessively large model structures still exist, and they cannot accurately distinguish between many similar scenes, hindering their deployment in service robots for practical applications. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a scene recognition method, terminal device, and storage medium based on knowledge graphs.

[0006] The specific plan is as follows:

[0007] A knowledge graph-based scene recognition method includes the following steps:

[0008] S1: Collect images of different types of scenes to form a training set, and label all target objects and their types in each image in the training set;

[0009] S2: Construct a scene target type recognition model and train the model using a training set;

[0010] The model's network structure includes an object detection module and a knowledge graph-based semantic analysis module;

[0011] The target detection module is an improvement on the YOLOv5 network. The backbone network of the YOLOv5 network is replaced with the MobileNet-V1 network, and the PANet structure of the YOLOv5 network is modified. The upsampling method in the FPN part of the PANet structure is replaced with subpixel convolution instead of nearest neighbor interpolation, and a 1×1 convolution kernel is added after each subpixel convolution.

[0012] After the target object detected by the target detection module is input into the semantic analysis module, the semantic analysis module analyzes the relationship between the target objects based on the pre-built scene knowledge graph and uses a random restart walk algorithm to obtain the relationship matrix between the target object and each entity in the scene knowledge graph.

[0013] The target type identification result is obtained by weighted summing the category matrix output by the target detection module and the relationship matrix output by the semantic analysis module.

[0014] S3: Use the trained model to identify the type of the target object in the image to be recognized;

[0015] S4: Calculate the membership degree of the image to be identified for each scene, and take the scene with the highest membership degree as the scene in which the image to be identified is located; the formula for calculating the membership degree is:

[0016] O j =X1*P1*S 1j +…+X i *P i *S ij +…+X n *P n *S nj

[0017] Among them, O j X represents the membership degree of the j-th scenario. i This indicates whether the i-th type of target object exists in the image to be identified. If it exists, X... i =1; if it does not exist, X i =0; P i S represents the confidence level of the detection result for the i-th target object; ijThis represents the probability that the i-th type of target object appears in the j-th scenario; i∈[1,n], where n represents the total number of target objects.

[0018] Furthermore, step S1 also includes performing data augmentation on the acquired images using the AdalN stylization algorithm before forming a training set.

[0019] Furthermore, the method for constructing the pre-built scene knowledge graph in step S2 is as follows: entity recognition is performed through BiLSTM-CRF, entity relationship extraction is performed through convolutional neural network, and entity attribute extraction is performed through web crawling combined with rule extraction.

[0020] Furthermore, the entity attributes of the target object include the weight of the target object relative to different scene types. The magnitude of the weight value is directly proportional to the probability of the target object appearing in the scene.

[0021] Furthermore, step S3, before using the trained model, also includes: converting the multi-branch structure in the MobileNet-V1 network of the model into a single-path structure.

[0022] Furthermore, the equivalent transformation method is as follows: by filling with zeros, the 1×1 convolution parallel to the 3×3 convolution is equivalently transformed into a 3×3 convolution, and then, based on the additivity of convolution, the two 3×3 convolutions are equivalently transformed into one 3×3 convolution.

[0023] A knowledge graph-based scene recognition terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the embodiments of the present invention.

[0024] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above in the embodiments of the present invention.

[0025] The present invention adopts the above technical solution, which improves the recognition accuracy while making the model lightweight and solving the problem of difficulty in distinguishing similar scenes. It is suitable for deployment on service robots for scene recognition. Attached Figure Description

[0026] Figure 1 The diagram shown is a flowchart of Embodiment 1 of the present invention.

[0027] Figure 2 The diagram shown is a schematic of the network structure of the model in this embodiment.

[0028] Figure 3 The diagram shown is an equivalent transformation diagram during model structure reconstruction in this embodiment.

[0029] Figure 4 The diagram shown is a schematic of the improved SP-PANet structure in this embodiment.

[0030] Figure 5 The image shown is a schematic diagram of the scene image recognition results in this embodiment. Detailed Implementation

[0031] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0032] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0033] Example 1:

[0034] This invention provides a scene recognition method based on knowledge graphs, such as... Figure 1 As shown, the method includes the following steps:

[0035] S1: Collect images of different types of scenes to form a training set, and label all target objects and their types in each image in the training set.

[0036] In this embodiment, some scene images were collected from open-source datasets on the internet, while others were acquired through image data collection in real-world scenarios to address the limitations of open-source datasets. The scene types used were mainly divided into four categories: classroom, conference room, library, and bookstore. Each scene type can be further subdivided into more specific types. Global and local images of different scene types were collected simultaneously. After cleaning out excessively low-quality (e.g., blurry) scene images, a total of 1485 images across the four scene types were obtained. Local image data of representative objects (such as blackboards, projectors, and bookshelves) in each scene were collected and used to supplement and improve the global scene images.

[0037] To prevent overfitting and other problems caused by insufficient scene image data in the trained model, this embodiment also includes expanding the processed scene images to enrich the scene image dataset. Specifically, the AdalN stylization algorithm is used to adjust the appropriate content ratio and style ratio during the stylization process. While ensuring that the high-level semantic information of the scene image remains unchanged, the texture, color and other features in the style map are transferred to the content map (scene image). New training samples are generated through data augmentation to expand the dataset.

[0038] In this embodiment, Labelimg annotation software is used to annotate the processed scene images. Rectangular bounding boxes are used to mark areas containing target object information within the image. Representative target objects and their types are labeled within the scene, enabling the subsequent object detection model to better learn more semantic and structural features from the image. The annotated files are saved as TXT files in YOLO format. The scene images are randomly allocated into training, validation, and test sets in a 7:2:1 ratio for use in training the object detection model.

[0039] S2: Construct an object detection model based on target relation knowledge learning, and train the model using a training set.

[0040] like Figure 2 As shown, the network structure of the model consists of two parts: an object detection module and a knowledge graph-based semantic analysis module.

[0041] 1. The target detection module has been improved based on the YOLOv5 network in the following ways:

[0042] (1) Replace the backbone network (feature extraction network) of YOLOv5 network with MobileNet-V1 network.

[0043] MobileNet-V1 is a lightweight network architecture. To further reduce its weight, this embodiment maintains the same network structure (i.e., a multi-branch structure) during training. During subsequent model deployment, the multi-branch structure is equivalently converted to a single-branch structure. This leverages both the high training performance of the multi-branch structure and the lightweight and fast performance of the single-branch structure. Specifically, the equivalent conversion first converts a 1×1 convolution parallel to a 3×3 convolution by padding with zeros. Then, based on the additivity of convolutions, two 3×3 convolutions are equivalently converted to one 3×3 convolution. Figure 3 As shown.

[0044] (2) Modify the PANet structure of the YOLOv5 network. In this embodiment, the modified PANet structure is named SP-PANet structure, and its structural diagram is shown below. Figure 4 As shown, specifically, the nearest neighbor interpolation upsampling method in the FPN part of the PANet structure, which destroys the semantic information of the feature map, is replaced with sub-pixel convolution upsampling, and the number of channels of the feature map is changed by adding a 1×1 convolution kernel. The low-resolution feature map of the original image is converted into a multi-channel feature map through convolution, and the feature map is reorganized to obtain a high-resolution feature map, which can provide more contextual information to help generate more realistic detailed features.

[0045] 2. After the target object detected by the target detection module is input into the semantic analysis module, the semantic analysis module analyzes the relationship between the target objects based on the pre-built scene knowledge graph and uses a random restart walk algorithm to obtain the relationship matrix between the target object and each entity in the scene knowledge graph.

[0046] The output of the object detection module is a category matrix. Where i represents the index of the detected target object, n represents the total number of detected target objects, j identifies the category index, m represents the total number of categories, and q ij This represents the probability that the i-th target object belongs to the j-th class.

[0047] The semantic analysis module outputs a relation matrix. Where, k ij This represents the strength of the relationship between the detected i-th target object and the j-th entity in the scene knowledge graph, where the j-th entity in the scene knowledge graph and the j-th category in the category matrix belong to the same object, such as both being cars.

[0048] The target type identification result is obtained by weighted summing of the category matrix P output by the target detection module and the relation matrix K output by the semantic analysis module.

[0049] The weighted sum of Q and K is the weighted sum of corresponding elements in the matrix. Let the matrix after the weighted sum be... but elements in σ1 and σ2 represent the weights of the category matrix and the relation matrix, respectively.

[0050] By adding the semantic analysis module in this embodiment, features can be enhanced based on the established relational or attribute knowledge in the ternary combination, building upon traditional detection methods. This approach focuses not only on the features of the target object itself in the scene image but also on the relationships between different target objects, achieving the effect of detecting target categories through common-sense reasoning abilities similar to humans.

[0051] The scene knowledge graph is constructed using third-party scene data. During its construction, a combination of BiLSTM-CRF model, convolutional neural network, and rule extraction transforms open information into triples (subject, relation, object), resulting in structured data. The scene knowledge graph is then built in Neo4j based on this structured data.

[0052] This paper utilizes a BiLSTM-CRF model to perform entity recognition. Each character is encoded as a character vector, which serves as the input to the BiLSTM. The predicted label for each character is then used as the input to the CRF layer. Next, constraints are added to the final predicted labels in the CRF layer, outputting the label sequence with the highest probability. These constraints reduce the number of invalid predicted label sequences. The BiLSTM, the first part of the model, consists of forward and backward LSTMs, and the outputs at the same time step are merged. The CRF, the second part of the model, is characterized by assuming that the output random variables form a Markov random field. In entity recognition, the CRF can compute an optimal predicted sequence by considering the relationships between adjacent labels. Adding this output compensates for the limitation of BiLSTM, which excels at handling information that is too far apart in text but cannot handle dependencies between adjacent labels.

[0053] Entity relation extraction is performed using a relation extraction model built on a Convolutional Neural Network (CNN). The CNN relation extraction model mainly consists of three parts: an input layer, a feature extraction layer, and an output layer. First, in the input layer, i.e., the word embedding layer, the neural network input is in vector form. For the input natural language sentence, after preprocessing such as word segmentation, each word in the sentence is converted into a low-dimensional vector. Second, in the feature extraction layer, a position vector is generated from the distance between each word and the two labeled entities, concatenated with the word vector, and then subjected to convolution and piecewise max pooling to obtain the sentence's feature vector. Finally, in the output layer, the sentence feature vector output from the feature extraction layer is input into a softmax classifier to obtain the final classification result.

[0054] Attribute extraction is performed using rule extraction. Based on the correct identification of text entities, entity attributes can be extracted from the information using web crawling combined with rule extraction. Different target objects have different probabilities of appearing in different scene types. Therefore, in the initially constructed scene knowledge graph, weight information for different target objects in different scene types is added to the entity attributes of the scene target objects. The magnitude of the weight value is directly proportional to the probability of the target object appearing in the scene.

[0055] When training with the training set, the model's weight biases are pre-trained, with a learning rate set to 0.001, 4 images input per pass, and 600 training generations. During training, the loss functions continuously decrease, while the model's precision, recall, and mean precision continuously increase, indicating a continuous improvement in model performance. The mean precision (mAP), the most common performance metric for object detection models, is used to objectively evaluate the model's performance. After training, the model's mAP is 80.5%, meeting the requirements.

[0056] S3: The trained model is used to identify the type of the target object in the image to be recognized.

[0057] S4: Calculate the membership degree of the image to be identified for each scene, and take the scene with the highest membership degree as the scene in which the image to be identified is located. The specific calculation process is as follows:

[0058] First, construct the target set X = {X1, ..., X...} i ,…,X n}, X i This indicates whether the i-th type of target object exists in the image to be identified. If it exists, X... i =1; if it does not exist, X i =0. i∈[1,n], where n represents the total number of target objects. The target set X corresponding to the image to be identified can be obtained through the recognition result of step S3.

[0059] Next, construct the confidence set P = {P1, ..., P2}. i ,…,P n}, P i This represents the confidence level of the detection result for the i-th type of target object. The value needs to be preset by those skilled in the art based on the actual situation.

[0060] Then construct the probability matrix S nk This represents the probability that the nth type of target object appears in the kth scenario. The value should correspond to the entity attributes of the target object in the knowledge graph.

[0061] Finally, construct the membership set O = {O1, ..., O2}. j ,…,O k}, O j Let represent the membership degree of the j-th scenario, and k represent the total number of scenarios. O j =X1*P1*S 1j +…+X i *P i *S ij +…+X n *P n *S nj Take the maximum value O in the membership set O. max O max The corresponding scene type is the scene type in which the scene image is located, and the recognition effect is as follows: Figure 4 As shown.

[0062] This invention addresses the shortcomings of current scene recognition algorithms, such as difficulty in distinguishing similar scenes and significant susceptibility to style influences. It combines mature object detection technologies with widely applicable knowledge graphs, and lightweights the model to construct a lightweight graph detection network based on target relationship knowledge learning. Taking into full account the varying probabilities of different target objects in different scenes and the challenges in recognizing similar scenes, a scene knowledge graph incorporating weighted information and common-sense factors is constructed. This combination ensures high scene recognition accuracy.

[0063] In this invention, the more complex scene recognition model is first converted into an object detection model when training the deep learning model, reducing the difficulty and time required for training the deep learning model and improving efficiency. Through data stylization processing, a lightweight graph detection network based on target relationship knowledge learning, and a scene knowledge graph containing weight information and common-sense factors, the problem of distinguishing similar scenes is further solved. This ensures recognition accuracy and speed while the lightweight network structure reduces the hardware cost required for deployment on service robots.

[0064] Example 2:

[0065] The present invention also provides a knowledge graph-based scene recognition terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the method embodiment described above in Embodiment 1 of the present invention.

[0066] Furthermore, as an executable solution, the knowledge graph-based scene recognition terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The knowledge graph-based scene recognition terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the knowledge graph-based scene recognition terminal device is merely an example and does not constitute a limitation on the knowledge graph-based scene recognition terminal device. It may include more or fewer components than described above, or combine certain components, or use different components. For example, the knowledge graph-based scene recognition terminal device may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0067] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices. The general-purpose processor can be a microprocessor or any conventional processor. This processor serves as the control center of the knowledge graph-based scene recognition terminal device, connecting all parts of the device via various interfaces and lines.

[0068] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the knowledge graph-based scene recognition terminal device by running or executing the computer programs and / or modules stored in the memory and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0069] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0070] If the modules / units integrated in the knowledge graph-based scene recognition terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0071] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A scene recognition method based on knowledge graphs, characterized in that, Includes the following steps: S1: Collect images of different types of scenes to form a training set, and label all target objects and their types in each image in the training set; S2: Construct a scene target type recognition model and train the model using a training set; The model's network structure includes an object detection module and a knowledge graph-based semantic analysis module; The target detection module is an improvement on the YOLOv5 network. The backbone network of the YOLOv5 network is replaced with the MobileNet-V1 network, and the PANet structure of the YOLOv5 network is modified. The upsampling method in the FPN part of the PANet structure is replaced with subpixel convolution instead of nearest neighbor interpolation, and a 1×1 convolution kernel is added after each subpixel convolution. After the target object detected by the target detection module is input into the semantic analysis module, the semantic analysis module analyzes the relationship between the target objects based on the pre-built scene knowledge graph and uses a random restart walk algorithm to obtain the relationship matrix between the target object and each entity in the scene knowledge graph. The pre-built scene knowledge graph is constructed by: performing entity recognition through BiLSTM-CRF, extracting entity relationships through convolutional neural networks, and extracting entity attributes through web crawling combined with rule extraction. The target type identification result is obtained by weighted summing the category matrix output by the target detection module and the relationship matrix output by the semantic analysis module. S3: Use the trained model to identify the type of the target object in the image to be recognized; S4: Calculate the membership degree of the image to be identified for each scene, and take the scene with the highest membership degree as the scene in which the image to be identified is located; the formula for calculating the membership degree is: Oh j =X1*p1*S 1j +…+X i *p i *S ij +…+X n *P n *S nj Among them, O j X represents the membership degree of the j-th scenario. i This indicates whether the i-th type of target object exists in the image to be identified. If it exists, X... i =1; if it does not exist, X i =0; P i S represents the confidence level of the detection result for the i-th target object; ij This represents the probability that the i-th type of target object appears in the j-th scenario; i∈[1,n], where n represents the total number of target objects.

2. The scene recognition method based on knowledge graphs according to claim 1, characterized in that: Step S1 also includes performing data augmentation on the acquired images using the AdalN stylization algorithm before forming a training set.

3. The scene recognition method based on knowledge graphs according to claim 1, characterized in that: The entity attributes of the target object include the weight of the target object relative to different scene types. The magnitude of the weight value is directly proportional to the probability of the target object appearing in the scene.

4. The scene recognition method based on knowledge graphs according to claim 1, characterized in that: Step S3, before using the trained model, also includes: converting the multi-branch structure in the MobileNet-V1 network of the model into a single-path structure.

5. The scene recognition method based on knowledge graphs according to claim 4, characterized in that: The equivalent transformation method is as follows: by filling with zeros, the 1×1 convolution parallel to the 3×3 convolution is equivalently transformed into a 3×3 convolution, and then, based on the additivity of convolution, the two 3×3 convolutions are equivalently transformed into one 3×3 convolution.

6. A scene recognition terminal device based on knowledge graph, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images