Knowledge graph application method and system based on deep model and real-time image processing

Abstract

The present invention provides a knowledge graph application method and system based on deep model and real-time image processing, the method comprising acquiring real-time image data; preprocessing the real-time image data; extracting features from the preprocessed image data using a large model, identifying entities, scene elements, and behavior predictions in the image; converting the identified entities and scene information into structured knowledge representations; associating the identified entities and relationships to construct a knowledge graph; continuously receiving newly input real-time image data, dynamically updating the knowledge graph, and re-evaluating and updating the relationships between entities through a reasoning mechanism; performing reasoning operations based on the dynamically updated knowledge graph to provide intelligent decision support for the autonomous driving system. The present invention combines large models and real-time image processing technology, and generates a dynamic knowledge graph based on natural language processing, providing intelligent decision support for application scenarios such as autonomous driving.

Description

Technical Field

[0001] The present invention relates to the field of automatic logistics AGV technology, and in particular to a knowledge graph application method based on deep model and real-time image processing. Background Art

[0002] With the rapid development of big data, artificial intelligence, and deep learning technologies, knowledge graphs, as a structured way of representing knowledge, have been widely used in many fields such as natural language processing, recommendation systems, and search engines. However, traditional knowledge graph construction methods mainly rely on manual annotation or rule extraction, which cannot efficiently cope with complex and changeable real-time data. On the other hand, with the rapid advancement of real-time image processing technology, obtaining information through real-time images and combining them with knowledge graphs for reasoning and application can greatly improve the system's automation level and the accuracy of information analysis.

[0003] Therefore, in the process of combining image processing and knowledge graphs, the existing technology has problems such as huge data volume, poor real-time performance, and inability to dynamically generate and update knowledge graphs. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a knowledge graph application method and system based on deep models and real-time image processing. The invention combines large models and real-time image processing technology, and generates dynamic knowledge graphs based on natural language processing, thereby realizing dynamic generation and updating of knowledge graphs and providing intelligent decision support for application scenarios such as autonomous driving.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] A knowledge graph application method based on a deep model and real-time image processing comprises the following steps:

[0007] Acquire real-time image data;

[0008] Preprocessing the real-time image data, the preprocessing steps including image resolution adjustment, denoising and data enhancement;

[0009] Use the pre-trained large model to extract features from the pre-processed image data, identify entities and scene elements in the image, and make behavior predictions;

[0010] Convert the identified entities and scene information into structured knowledge representation;

[0011] Associate the identified entities and relationships to build a real-time, local knowledge graph;

[0012] Continuously receive new input real-time image data, dynamically update the knowledge graph based on the new input real-time image data, and re-evaluate and update the relationship between entities through the reasoning mechanism;

[0013] Perform reasoning operations based on dynamically updated knowledge graphs to provide intelligent decision support for autonomous driving systems.

[0014] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, during the driving process of a vehicle, multiple cameras installed around the vehicle body are used to continuously capture real-time image data of the surrounding environment, wherein the cameras include at least one of a wide-angle camera, a night vision camera or a thermal imaging camera, so as to ensure that clear images can be obtained under various lighting conditions and environments;

[0015] The captured real-time image data is transmitted to the vehicle-mounted computing system, and the real-time image processing module receives and processes the image data.

[0016] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the preprocessing step specifically includes:

[0017] Image resolution adjustment: Standardize the resolution of captured real-time image data to ensure consistency of data input to subsequent processing modules;

[0018] Denoising: Use Gaussian blur, median filtering or other denoising algorithms to process image data to remove noise in the image and improve image quality;

[0019] Data enhancement: Rotate, flip, and scale the preprocessed image data to adapt it to different camera angles, lighting conditions, and complex and changing traffic environments.

[0020] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the method uses a pre-trained large model to extract features from pre-processed image data, including:

[0021] Input the preprocessed image data into a pre-trained large model based on a convolutional neural network;

[0022] Through the feature layer of the large model, key information in the image is extracted, including:

[0023] Entity recognition: Use the CNN model to identify traffic participants in the image and further extract the specific features of each traffic participant;

[0024] Scene understanding: The CNN model is used to identify scene elements in the image, providing the necessary contextual information and environmental perception for vehicle decision-making;

[0025] Behavior prediction: Analyze the motion information of other vehicles and pedestrians in the image based on the CNN model to predict the next behavior.

[0026] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, after identifying the entity and scene information in the image, the following steps are performed to generate a structured knowledge representation, including:

[0027] Entity node construction: convert each identified entity into an entity node in the knowledge graph and assign a unique identifier to each entity node;

[0028] Attribute assignment: For each entity node, its specific attributes are assigned as the attributes of the node to form a complete entity description;

[0029] Establishing relationship edges: Based on the spatial position relationship, interaction behavior, and scene understanding results between entities, relationship edges between entity nodes are constructed to represent the association and interaction between entities.

[0030] Generation of structured knowledge representation: Through the above-mentioned process of entity node construction, attribute assignment and relationship edge establishment, the entity and scene information in the image is converted into a structured knowledge representation, forming a knowledge graph containing entity nodes, attribute information and relationship edges, thereby providing structured data support.

[0031] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the construction of the knowledge graph includes:

[0032] Entity and relationship association: Dynamically associate the identified entities with the relationships between them, where entities are nodes in the knowledge graph and relationships are edges connecting nodes;

[0033] Real-time local knowledge graph construction: Based on the associated entities and relationships, a real-time, local knowledge graph is constructed. The graph is used to represent the current traffic scene of the vehicle and contains all entities directly related to the vehicle's travel and the relationships between them.

[0034] Dynamic update of the knowledge graph: As new real-time image data is continuously input, the nodes and edges in the knowledge graph are continuously updated to reflect changes in the traffic environment around the vehicle. At the same time, entities and relationships that are no longer valid or have changed are removed or modified to ensure the real-time and accuracy of the knowledge graph.

[0035] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the knowledge graph specifically includes:

[0036] Entity nodes representing specific traffic participants and scene elements. Each node represents an independent entity and has a unique identifier.

[0037] Assign specific entity attribute information to each entity node, which is dynamically updated based on real-time image data;

[0038] Entity relationships, the entity nodes in the graph are connected by relationship edges, which represent their spatial position relationships, interaction states, or scene logic.

[0039] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the knowledge graph is dynamically updated according to the newly input real-time image data, including:

[0040] Real-time update of image recognition results: Continuously receive new image data from vehicle-mounted cameras or other image acquisition devices, and use image recognition technology to re-identify the status of traffic participants; when a change in entity status is detected, immediately update the change to the corresponding entity node attributes in the knowledge graph;

[0041] Dynamic re-evaluation of relationships: As the vehicle moves and the traffic environment changes, the relationship between entities is re-evaluated in real time using the reasoning mechanism. When the distance between the car and the pedestrian is detected to be reduced below a certain threshold, the reasoning mechanism is used to add a marker for potential collision risk, and a warning signal is issued accordingly, or the behavior strategy of the autonomous driving system is adjusted;

[0042] Incremental update mechanism: In each update, only the changed parts are updated instead of rebuilding the entire knowledge graph, including identifying and updating the changed entity attributes, adding or deleting relationship edges, and updating the inference results.

[0043] According to a knowledge graph application method based on a deep model and real-time image processing provided by the present invention, the reasoning operation based on the dynamically updated knowledge graph includes:

[0044] When pedestrians are identified in front through real-time image processing and the distance, speed and relative position between the vehicle and the pedestrians are judged to constitute a potential collision risk based on the entity relationship in the knowledge graph, a recommendation for deceleration or braking is inferred and provided as a decision input to the vehicle's control system or driver assistance system to achieve timely safety response;

[0045] When a state change of a traffic light is detected, the system infers that the vehicle needs to stop and wait based on the relationship between the traffic light and the vehicle's current road in the knowledge graph, as well as the constraints of traffic regulations. The system then converts this inference result into control instructions or driving suggestions to ensure that the vehicle complies with traffic regulations and stops safely.

[0046] A knowledge graph application system based on deep model and real-time image processing, comprising:

[0047] A real-time image acquisition module, used for acquiring real-time image data;

[0048] An image data preprocessing module, used to preprocess the real-time image data, wherein the preprocessing steps include image resolution adjustment, denoising and data enhancement;

[0049] The image recognition and feature extraction module uses a pre-trained large model to extract features from pre-processed image data, identify entities and scene elements in the image, and make behavior predictions;

[0050] The structured representation module is used to transform the identified entities and scene information into structured knowledge representation;

[0051] The knowledge graph preliminary construction module is used to associate the identified entities and relationships to build a real-time, local knowledge graph;

[0052] The dynamic update and reasoning module is used to continuously receive new input real-time image data, dynamically update the knowledge graph based on the new input real-time image data, and re-evaluate and update the relationship between entities through the reasoning mechanism;

[0053] The driving decision module performs reasoning operations based on the dynamically updated knowledge graph to provide intelligent decision support for the autonomous driving system.

[0054] It can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. Combination of real-time image processing and knowledge extraction: The present invention combines real-time image processing technology based on large models with knowledge graph construction technology to achieve automatic extraction of knowledge from visual data and generate structured knowledge graphs, greatly improving the degree of automation and real-time performance of information extraction.

[0056] 2. Dynamic knowledge graph construction and update: The present invention designs a dynamic knowledge graph update mechanism so that the knowledge graph can be automatically updated and adaptively optimized based on the input of real-time image data to adapt to changing and complex environmental requirements.

[0057] 3. Data processing speed and accuracy: The present invention uses a pre-trained large model for feature extraction, which can efficiently extract key information from complex images, such as entities, scene elements, and behavior predictions, thereby improving data processing speed and accuracy.

[0058] 4. Structured knowledge representation: The present invention converts the identified entity and scene information into structured knowledge representation, which facilitates the storage, retrieval and reasoning of knowledge and enhances the availability and understandability of knowledge.

[0059] 5. Intelligent reasoning and decision support: The present invention is based on a dynamically updated knowledge graph. By combining the reasoning capabilities of a large model, the system can perform complex reasoning operations and provide intelligent decision support for the autonomous driving system, such as path planning and obstacle avoidance, thereby improving the safety and rationality of driving.

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of an embodiment of a knowledge graph application method based on a deep model and real-time image processing of the present invention.

[0062] Figure 2 It is a schematic diagram of the principle of inputting image data into generating a knowledge graph in an embodiment of a knowledge graph application method based on a deep model and real-time image processing of the present invention.

[0063] Figure 3 It is a schematic diagram of the principle of how to update the existing knowledge graph according to the newly input real-time image data in an embodiment of the knowledge graph application method based on the deep model and real-time image processing of the present invention.

[0064] Figure 4 It is a schematic diagram of an embodiment of a knowledge graph application system based on a deep model and real-time image processing of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] Regarding the real-time nature of image information and knowledge extraction: Existing knowledge graphs mainly rely on text information construction. The real-time and diversity of image information are difficult to be effectively utilized. This embodiment solves the problem of how to combine real-time image processing with dynamic construction of knowledge graphs.

[0068] Regarding dynamic updating and reasoning: Most existing knowledge graphs are static or quasi-static and lack a dynamic updating mechanism. This embodiment provides a knowledge graph dynamic updating and adaptive reasoning mechanism to ensure that the knowledge graph can reflect new information in a timely manner.

[0069] Regarding system integration and versatility: It is difficult for existing technologies to uniformly process multimodal information (such as text, images, videos, etc.). This embodiment proposes a multimodal integration method to make the system more versatile and able to adapt to application scenarios in different fields.

[0070] A knowledge graph application method embodiment based on deep model and real-time image processing

[0071] See also Figures 1 to 3 The present invention provides a knowledge graph application method based on a deep model and real-time image processing, the method comprising the following steps:

[0072] Step S1, acquiring real-time image data;

[0073] Step S2, preprocessing the real-time image data, the preprocessing steps include image resolution adjustment, denoising and data enhancement;

[0074] Step S3, using the pre-trained large model to extract features from the pre-processed image data, identify entities and scene elements in the image, and perform behavior prediction;

[0075] Step S4, converting the identified entity and scene information into structured knowledge representation;

[0076] Step S5, associating the identified entities and relationships to build a real-time, local knowledge graph;

[0077] Step S6, continuously receiving newly input real-time image data, dynamically updating the knowledge graph according to the newly input real-time image data, and re-evaluating and updating the relationship between entities through the reasoning mechanism;

[0078] Step S7, performing reasoning operations based on the dynamically updated knowledge graph to provide intelligent decision support for the autonomous driving system.

[0079] In the above step S1, while the vehicle is driving, multiple cameras installed around the vehicle body (front, back, left, and right) are used to continuously capture real-time image data of the surrounding environment, wherein the cameras include at least one of a wide-angle camera, a night vision camera, or a thermal imaging camera to ensure that clear images can be obtained under various lighting conditions and environments.

[0080] The captured real-time image data is transmitted to the vehicle-mounted computing system, and the real-time image processing module receives and processes the image data.

[0081] In the above step S2, the preprocessing step specifically includes:

[0082] Image resolution adjustment: Standardize the resolution of captured real-time image data to ensure consistency of data input to subsequent processing modules;

[0083] Denoising: Use Gaussian blur, median filtering or other denoising algorithms to process image data to remove noise in the image and improve image quality;

[0084] Data enhancement: Perform operations such as rotation, flipping, and scaling on the preprocessed image data to enhance the robustness of the model and enable it to adapt to different camera angles, lighting conditions, and complex and changing traffic environments.

[0085] In the above step S3, the pre-trained large model is used to extract features from the pre-processed image data, including:

[0086] Input the preprocessed image data into a pre-trained large model based on a convolutional neural network;

[0087] Through the feature layer of the large model, key information in the image is extracted, including:

[0088] Entity recognition: Use the CNN model to identify traffic participants in the image (such as vehicles, pedestrians, bicycles, etc.), and further extract the specific features of each traffic participant (such as vehicle type, color, etc.);

[0089] Scene understanding: The CNN model can be used to identify scene elements such as traffic signs, signal lights, lane lines, and road conditions in images, providing the necessary contextual information and environmental perception for vehicle decision-making;

[0090] Behavior prediction: Analyze the motion information of other vehicles and pedestrians in the image based on the CNN model to predict the next behavior (such as turning, braking, changing lanes, etc.).

[0091] In the above step S4, after identifying the entity and scene information in the image, the following steps are performed to generate a structured knowledge representation, including:

[0092] Entity node construction: convert each identified entity into an entity node in the knowledge graph and assign a unique identifier to each entity node;

[0093] Attribute assignment: For each entity node, its specific attributes are assigned as the attributes of the node to form a complete entity description;

[0094] Establishing relationship edges: Based on the spatial position relationship, interaction behavior, and scene understanding results between entities, relationship edges between entity nodes are constructed to represent the association and interaction between entities.

[0095] Generation of structured knowledge representation: Through the above-mentioned process of entity node construction, attribute assignment and relationship edge establishment, the entity and scene information in the image is converted into a structured knowledge representation, forming a knowledge graph containing entity nodes, attribute information and relationship edges, thereby providing structured data support.

[0096] It can be seen that after identifying the entity and scene information, the system converts it into a structured knowledge representation. For example, the car in the image is identified as an "entity node", and its color, speed, direction and other attributes are used as the attributes of the node; the relationship between the car and other entities (such as roads, pedestrians) is identified as a "relationship edge", such as "car-driving-on-road", "car-approaching-pedestrians", etc.

[0097] In the above step S5, the construction of the knowledge graph includes:

[0098] Entity and relationship association: Dynamically associate the identified entities with the relationships between them, where entities are nodes in the knowledge graph and relationships are edges connecting nodes;

[0099] Real-time local knowledge graph construction: Based on the associated entities and relationships, a real-time, local knowledge graph is constructed. The graph is used to represent the current traffic scene of the vehicle and contains all entities directly related to the vehicle's travel and the relationships between them.

[0100] Dynamic update of the knowledge graph: As new real-time image data is continuously input, the nodes and edges in the knowledge graph are continuously updated to reflect changes in the traffic environment around the vehicle. At the same time, entities and relationships that are no longer valid or have changed are removed or modified to ensure the real-time and accuracy of the knowledge graph.

[0101] In this embodiment, the knowledge graph specifically includes:

[0102] Entity nodes representing specific traffic participants and scene elements, each node represents an independent entity and has a unique identifier, for example, car 1, pedestrian A, traffic light, road R1.

[0103] Each entity node is assigned specific entity attribute information, which is dynamically updated according to real-time image data. For example, the speed of car 1 is 60km / h and the distance of pedestrian A is 20 meters.

[0104] Entity relationships: Entity nodes in the graph are connected by relationship edges, which represent their spatial position relationship, interaction status, or scene logic. For example, car 1 is driving on road R1, car 1 is approaching pedestrian A, and the traffic light is red.

[0105] In the above step S6, the knowledge graph is dynamically updated according to the newly input real-time image data, including:

[0106] Real-time update of image recognition results: Continuously receive new image data from vehicle-mounted cameras or other image acquisition devices, and use image recognition technology to re-identify the status of traffic participants; when a change in entity status is detected, immediately update the change to the corresponding entity node attribute in the knowledge graph. For example, when a traffic light is detected to change from green to red, the status of the light in the knowledge graph will be automatically updated.

[0107] Dynamic re-evaluation of relationships: As the vehicle moves and the traffic environment changes, the relationship between entities is re-evaluated in real time using the reasoning mechanism; when the distance between the car and the pedestrian is detected to be reduced below a certain threshold, the reasoning mechanism is used to increase the mark of potential collision risk, and accordingly issue a warning signal or adjust the behavior strategy of the autonomous driving system. For example, if the distance between the car and the pedestrian is detected to be reduced, the system will increase the reasoning result of "potential collision risk" and issue a warning to the driver.

[0108] Incremental update mechanism: In each update, only the changed parts are updated, rather than rebuilding the entire knowledge graph, including identifying and updating the changed entity attributes, adding or deleting relationship edges, and updating the inference results. It can be seen that the system adopts an incremental update method, updating only the changed parts to reduce computing overhead and ensure the real-time performance of the system.

[0109] In the above step S7, the reasoning operation is performed based on the dynamically updated knowledge graph, including:

[0110] When a pedestrian is identified in front through real-time image processing and the distance, speed, and relative position between the vehicle and the pedestrian are judged to constitute a potential collision risk based on the entity relationship in the knowledge graph, a suggestion for slowing down or braking is inferred, and this suggestion is provided as a decision input to the vehicle's control system or driver assistance system to achieve a timely safety response. For example, when the system detects a pedestrian in front, the knowledge graph will infer a suggestion of "need to slow down" or "brake".

[0111] When a traffic light status change is detected, the system infers that the vehicle needs to stop and wait based on the relationship between the traffic light and the vehicle's current road in the knowledge graph and the constraints of traffic regulations. This inference result is converted into a control instruction or driving suggestion to ensure that the vehicle complies with traffic regulations and stops safely. When the traffic light turns red, the system infers that the vehicle needs to "stop and wait".

[0112] Therefore, through the above methods, strong decision-making support can be provided for autonomous driving, ensuring that the vehicle makes safe and reasonable decisions in complex traffic environments.

[0113] An embodiment of a knowledge graph application system based on deep model and real-time image processing

[0114] like Figure 4 As shown, this embodiment provides a knowledge graph application system based on a deep model and real-time image processing, including:

[0115] A real-time image acquisition module 1 is used to acquire real-time image data;

[0116] Image data preprocessing module 2, used for preprocessing real-time image data, the preprocessing steps include image resolution adjustment, denoising and data enhancement;

[0117] Image recognition and feature extraction module 3 uses a pre-trained large model to extract features from pre-processed image data, identify entities and scene elements in the image, and perform behavior prediction;

[0118] The structured representation module 4 is used to transform the identified entity and scene information into structured knowledge representation;

[0119] The knowledge graph preliminary construction module 5 is used to associate the identified entities and relationships to build a real-time, local knowledge graph;

[0120] Dynamic update and reasoning module 6, used to continuously receive newly input real-time image data, dynamically update the knowledge graph according to the newly input real-time image data, and re-evaluate and update the relationship between entities through the reasoning mechanism;

[0121] The driving decision module 7 performs reasoning operations based on the dynamically updated knowledge graph to provide intelligent decision support for the autonomous driving system.

[0122] In the real-time image acquisition module 1, during the driving process of the vehicle, multiple cameras installed around the vehicle body are used to continuously capture real-time image data of the surrounding environment, wherein the camera includes at least one of a wide-angle camera, a night vision camera or a thermal imaging camera to ensure that clear images can be obtained under various lighting conditions and environments. The captured real-time image data is transmitted to the on-board computing system, and the real-time image processing module receives and processes the image data.

[0123] In the above-mentioned image data preprocessing module 2, it specifically includes:

[0124] Image resolution adjustment module: standardizes the resolution of captured real-time image data to ensure the consistency of data input to subsequent processing modules;

[0125] De-noising processing module: Use Gaussian blur, median filtering or other denoising algorithms to process image data to remove noise in the image and improve image quality;

[0126] Data enhancement module: Perform operations such as rotation, flipping, and scaling on the preprocessed image data to enhance the robustness of the model and enable it to adapt to different camera angles, lighting conditions, and complex and changing traffic environments.

[0127] In the above-mentioned image recognition and feature extraction module 3, it includes:

[0128] Input the preprocessed image data into a pre-trained large model based on a convolutional neural network;

[0129] Through the feature layer of the large model, key information in the image is extracted, including:

[0130] Entity recognition module: Use the CNN model to identify traffic participants in the image and further extract the specific features of each traffic participant;

[0131] Scene understanding module: uses the CNN model to identify scene elements such as traffic signs, signal lights, lane lines, road conditions, etc. in the image, providing necessary contextual information and environmental perception for vehicle decision-making;

[0132] Behavior prediction module: Analyzes the motion information of other vehicles and pedestrians in the image based on the CNN model and predicts the next behavior.

[0133] In the above structured representation module 4, it includes:

[0134] Entity node construction module: converts each identified entity into an entity node in the knowledge graph and assigns a unique identifier to each entity node;

[0135] Attribute assignment module: for each entity node, assign its specific attributes as the attributes of the node to form a complete entity description;

[0136] Relationship edge building module: Based on the spatial position relationship, interaction behavior and scene understanding results between entities, the relationship edges between entity nodes are constructed to represent the association and interaction between entities;

[0137] Structured knowledge representation generation module: Through the above-mentioned process of entity node construction, attribute assignment and relationship edge establishment, the entity and scene information in the image is converted into structured knowledge representation, forming a knowledge graph containing entity nodes, attribute information and relationship edges, thereby providing structured data support.

[0138] In the above-mentioned knowledge graph preliminary construction module 5, the construction of the knowledge graph includes:

[0139] Entity and relationship association module: dynamically associates the identified entities with the relationships between them, where entities are nodes in the knowledge graph and relationships are edges connecting nodes;

[0140] Real-time local knowledge graph construction module: Based on the associated entities and relationships, a real-time, local knowledge graph is constructed. The graph is used to represent the current traffic scene of the vehicle and contains all entities directly related to the vehicle's travel and the relationships between them.

[0141] Knowledge graph dynamic update module: As new real-time image data is continuously input, the nodes and edges in the knowledge graph are continuously updated to reflect changes in the traffic environment around the vehicle. At the same time, entities and relationships that are no longer valid or have changed are removed or modified to ensure the real-time and accuracy of the knowledge graph.

[0142] In the above dynamic update and reasoning module 6, it includes:

[0143] Real-time update module of image recognition results: continuously receives new image data from vehicle-mounted cameras or other image acquisition devices, and uses image recognition technology to re-identify the status of traffic participants; when a change in entity status is detected, this change is immediately updated to the corresponding entity node attributes in the knowledge graph.

[0144] Dynamic re-evaluation module of relationships: As the vehicle moves and the traffic environment changes, the relationship between entities is re-evaluated in real time using the reasoning mechanism. When it is detected that the distance between the car and the pedestrian decreases below a certain threshold, the reasoning mechanism is used to increase the mark of potential collision risk, and accordingly issue a warning signal or adjust the behavior strategy of the autonomous driving system.

[0145] Incremental update mechanism module: In each update, only the changed parts are updated instead of rebuilding the entire knowledge graph, including identifying and updating the changed entity attributes, adding or deleting relationship edges, and updating the reasoning results.

[0146] In the above driving decision module 7, it includes:

[0147] When pedestrians are identified ahead through real-time image processing and the distance, speed, and relative position between the vehicle and the pedestrians are judged to constitute a potential collision risk based on the entity relationship in the knowledge graph, a recommendation for deceleration or braking is inferred and provided as a decision input to the vehicle's control system or driver assistance system to achieve timely safety response.

[0148] When a state change of a traffic light is detected, the system infers that the vehicle needs to stop and wait based on the relationship between the traffic light and the vehicle's current road in the knowledge graph, as well as the constraints of traffic regulations. The system then converts this inference result into control instructions or driving suggestions to ensure that the vehicle complies with traffic regulations and stops safely.

[0149] In summary, this embodiment combines the real-time image processing technology based on a large model with the knowledge graph construction technology to achieve automatic extraction of knowledge from visual data and generate a structured knowledge graph, which greatly improves the automation and real-time performance of information extraction.

[0150] Furthermore, this embodiment designs a dynamic update mechanism for the knowledge graph, so that the knowledge graph can automatically perform incremental updates and adaptive optimization based on the input of real-time image data to adapt to changing and complex environmental requirements.

[0151] Furthermore, this embodiment utilizes a pre-trained large model for feature extraction, which can efficiently extract key information from complex images, such as entities, scene elements, and behavior predictions, thereby improving data processing speed and accuracy.

[0152] Furthermore, this embodiment converts the identified entity and scene information into a structured knowledge representation, which facilitates the storage, retrieval and reasoning of knowledge and enhances the availability and comprehensibility of knowledge.

[0153] Furthermore, this embodiment is based on a dynamically updated knowledge graph, and by combining the reasoning capabilities of a large model, the system can perform complex reasoning operations and provide intelligent decision support for the autonomous driving system, such as path planning and obstacle avoidance, thereby improving the safety and rationality of driving.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A knowledge graph application method based on deep model and real-time image processing, characterized in that: The following steps are involved: Acquire real-time image data; Preprocessing the real-time image data, the preprocessing steps including image resolution adjustment, denoising and data enhancement; Use the pre-trained large model to extract features from the pre-processed image data, identify entities and scene elements in the image, and make behavior predictions; Convert the identified entities and scene information into structured knowledge representation; Associate the identified entities and relationships to build a real-time, local knowledge graph; Continuously receive new input real-time image data, dynamically update the knowledge graph based on the new input real-time image data, and re-evaluate and update the relationship between entities through the reasoning mechanism; Perform reasoning operations based on dynamically updated knowledge graphs to provide intelligent decision support for autonomous driving systems.

2. The method according to claim 1, characterized in that: During the driving process of the vehicle, multiple cameras installed around the vehicle body are used to continuously capture real-time image data of the surrounding environment, wherein the cameras include at least one of a wide-angle camera, a night vision camera or a thermal imaging camera to ensure that clear images can be obtained under various lighting conditions and environments; The captured real-time image data is transmitted to the vehicle-mounted computing system, and the real-time image processing module receives and processes the image data.

3. The method according to claim 1, characterized in that The pre-processing step specifically includes: Image resolution adjustment: Standardize the resolution of captured real-time image data to ensure consistency of data input to subsequent processing modules; Denoising: Use Gaussian blur, median filtering or other denoising algorithms to process image data to remove noise in the image and improve image quality; Data enhancement: Rotate, flip, and scale the preprocessed image data to adapt it to different camera angles, lighting conditions, and complex and changing traffic environments.

4. The method according to claim 1, characterized in that The method of extracting features from the preprocessed image data using the pretrained large model includes: Input the preprocessed image data into a pre-trained large model based on a convolutional neural network; Through the feature layer of the large model, key information in the image is extracted, including: Entity recognition: Use the CNN model to identify traffic participants in the image and further extract the specific features of each traffic participant; Scene understanding: The CNN model is used to identify scene elements in the image, providing the necessary contextual information and environmental perception for vehicle decision-making; Behavior prediction: Analyze the motion information of other vehicles and pedestrians in the image based on the CNN model to predict the next behavior.

5. The method according to claim 1, characterized in that: After identifying the entities and scene information in the image, the following steps are performed to generate a structured knowledge representation, including: Entity node construction: convert each identified entity into an entity node in the knowledge graph and assign a unique identifier to each entity node; Attribute assignment: For each entity node, its specific attributes are assigned as the attributes of the node to form a complete entity description; Establishing relationship edges: Based on the spatial position relationship, interaction behavior, and scene understanding results between entities, relationship edges between entity nodes are constructed to represent the association and interaction between entities. Generation of structured knowledge representation: Through the above-mentioned process of entity node construction, attribute assignment and relationship edge establishment, the entity and scene information in the image is converted into a structured knowledge representation, forming a knowledge graph containing entity nodes, attribute information and relationship edges, thereby providing structured data support.

6. The method according to claim 1, characterized in that The construction of the knowledge graph includes: Entity and relationship association: Dynamically associate the identified entities with the relationships between them, where entities are nodes in the knowledge graph and relationships are edges connecting nodes; Real-time local knowledge graph construction: Based on the associated entities and relationships, a real-time, local knowledge graph is constructed. The graph is used to represent the current traffic scene of the vehicle and contains all entities directly related to the vehicle's travel and the relationships between them. Dynamic update of the knowledge graph: As new real-time image data is continuously input, the nodes and edges in the knowledge graph are continuously updated to reflect changes in the traffic environment around the vehicle. At the same time, entities and relationships that are no longer valid or have changed are removed or modified to ensure the real-time and accuracy of the knowledge graph.

7. The method according to claim 6, characterized in that: The knowledge graph specifically includes: Entity nodes representing specific traffic participants and scene elements. Each node represents an independent entity and has a unique identifier. Assign specific entity attribute information to each entity node, which is dynamically updated based on real-time image data; Entity relationships, the entity nodes in the graph are connected by relationship edges, which represent their spatial position relationships, interaction states, or scene logic.

8. The method according to any one of claims 1 to 7, characterized in that: The dynamically updating the knowledge graph according to the newly input real-time image data includes: Real-time update of image recognition results: Continuously receive new image data from vehicle-mounted cameras or other image acquisition devices, and use image recognition technology to re-identify the status of traffic participants; when a change in entity status is detected, immediately update the change to the corresponding entity node attributes in the knowledge graph; Dynamic re-evaluation of relationships: As the vehicle moves and the traffic environment changes, the relationship between entities is re-evaluated in real time using the reasoning mechanism. When the distance between the car and the pedestrian is detected to be reduced below a certain threshold, the reasoning mechanism is used to add a marker for potential collision risk, and a warning signal is issued accordingly, or the behavior strategy of the autonomous driving system is adjusted; Incremental update mechanism: In each update, only the changed parts are updated instead of rebuilding the entire knowledge graph, including identifying and updating the changed entity attributes, adding or deleting relationship edges, and updating the inference results.

9. The method according to any one of claims 1 to 7, characterized in that: The performing of reasoning operations based on the dynamically updated knowledge graph includes: When pedestrians are identified in front through real-time image processing and the distance, speed and relative position between the vehicle and the pedestrians are judged to constitute a potential collision risk based on the entity relationship in the knowledge graph, a recommendation for deceleration or braking is inferred and provided as a decision input to the vehicle's control system or driver assistance system to achieve timely safety response; When a state change of a traffic light is detected, the system infers that the vehicle needs to stop and wait based on the relationship between the traffic light and the vehicle's current road in the knowledge graph, as well as the constraints of traffic regulations. The system then converts this inference result into control instructions or driving suggestions to ensure that the vehicle complies with traffic regulations and stops safely.

10. A knowledge graph application system based on deep model and real-time image processing, characterized in that: include: A real-time image acquisition module, used for acquiring real-time image data; An image data preprocessing module, used to preprocess the real-time image data, wherein the preprocessing steps include image resolution adjustment, denoising and data enhancement; The image recognition and feature extraction module uses a pre-trained large model to extract features from pre-processed image data, identify entities and scene elements in the image, and make behavior predictions; The structured representation module is used to transform the identified entities and scene information into structured knowledge representation; The knowledge graph preliminary construction module is used to associate the identified entities and relationships to build a real-time, local knowledge graph; The dynamic update and reasoning module is used to continuously receive new input real-time image data, dynamically update the knowledge graph based on the new input real-time image data, and re-evaluate and update the relationship between entities through the reasoning mechanism; The driving decision module performs reasoning operations based on the dynamically updated knowledge graph to provide intelligent decision support for the autonomous driving system.

Images