Peripheral environment risk source intelligent identification method and system based on virtual sample set
By building a digital twin scenario to generate a virtual sample set and training the corresponding neural network, the problem of low efficiency in identifying risk sources around the railway is solved, and efficient and accurate risk source identification and change area monitoring are achieved.
Patent Information
- Application Number
- CN202311116754.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-31
- Publication Date
- 2025-02-28
AI Technical Summary
The identification efficiency of environmental risk sources around railways is low, and it is difficult for the prior art to effectively identify multiple categories of risk sources and their changing areas, especially when the sample set is insufficient and the quality of image labeling is uneven.
Using an intelligent recognition method based on virtual sample sets, a virtual sample set is generated by building a digital twin scenario, and the Yolov5 network and a fully convolutional neural network are trained to achieve the identification and matching of risk-hit objects and changing areas.
It improves the efficiency and accuracy of risk source identification, reduces the deviation and workload of manual labeling, enhances the generalization ability of the model, and can effectively identify multiple categories of risk sources and their changing areas.
Smart Images
Figure CN119540845A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of environmental risk source identification, and in particular relates to a method and system for intelligently identifying surrounding environmental risk sources based on a virtual sample set. Background Art
[0002] The external environmental risk sources of railways have become the main target of railway safety problem rectification work. Doing a good job in the rectification of external environmental safety problems of railways is an important means to prevent and resolve major railway safety hazards and ensure the safety of high-speed railways and passenger trains. At the same time, it also puts forward higher requirements for the detection, identification and disposal of external risk sources of railways.
[0003] According to the "Railway External Environment Safety Management Measures", railway external environment hazards can be divided into 18 categories, including illegal construction, dangerous goods, and parallel crossings. Typical risk source hazards include color steel tile houses, dust nets, plastic films, and dangerous trees. The discovery and inspection of these risk sources mainly rely on three means: "human defense, technical defense, and physical defense." Considering that railways are linear projects, multiple types of risk hazards and complex external environments are mutually infiltrated. In technical defense, means such as remote sensing satellites and drones equipped with cameras are playing an increasingly important role.
[0004] Technical personnel mainly rely on manual browsing and viewing of satellite remote sensing data and drone inspection video images. Although advanced technical defense means ensure the accessibility of the inspection range, the manual identification of risk hazards is inefficient, the work intensity is high, and the change area is difficult to identify. With the rapid development of image recognition technology based on computer vision, the automatic identification algorithm of risk sources is also being studied in depth. The implementation of image recognition technology requires two conditions: one is sufficient sample sets in different scenarios; the other is the accuracy and reliability of the intelligent recognition algorithm, that is, the detection rate. Due to the complex environment around the railway, there are many types of risk sources, and the forms (such as spatial position, angle, seasonal changes, etc.) are different, making it difficult to collect sample sets of different categories, and there are few training sample sets. At the same time, due to the influence of human factors in image annotation, the quality of manual annotation is uneven, and the workload is large, resulting in a certain deviation in the manually annotated samples; at the same time, the external environment involves the identification of hidden danger categories and change areas. At present, most of the target detection methods cannot effectively meet the above requirements. Therefore, it is necessary to propose a new method to solve the problem of small training sample sets and coordinate the accuracy and comprehensiveness of the recognition algorithm. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the prior art and propose a method and system for intelligently identifying surrounding environment risk sources based on a virtual sample set.
[0006] In order to achieve the above object, the present invention proposes a method for intelligently identifying surrounding environment risk sources based on a virtual sample set, the method comprising:
[0007] The video image of the environment to be detected is input into the pre-established and trained risk hidden danger object detection model and SCD change area detection model respectively, and the risk hidden danger object boundary box set and the change area set are detected respectively;
[0008] Based on the similarity algorithm, the information of the risk potential object boundary box and the change area is matched and the change situation is tracked, thereby realizing the identification of the risk source of the surrounding environment;
[0009] The risk hidden danger object detection model adopts the Yolov5 network, and the SCD change area detection model adopts a fully convolutional neural network. By constructing a digital twin scene to generate a virtual sample set, the risk hidden danger object detection model and the SCD (SceneChange Detection) change area detection model are trained.
[0010] As an improvement of the above method, generating a virtual sample set by constructing a digital twin scene specifically includes:
[0011] Step 1) Create a digital twin scene based on Unreal Engine UE4;
[0012] Step 2) Render the target samples in layers in the digital twin scene, segment them with different colors, obtain the target mask image, automatically annotate the target object bounding box based on the target mask image, and generate a virtual sample set corresponding to the target task.
[0013] As an improvement of the above method, the digital twin scene in step 1) includes: terrain, oblique photography, scene model, BIM model, vegetation, vehicles, lighting and visual effects.
[0014] As an improvement of the above method, the step 1) specifically includes:
[0015] Obtain terrain image data through the water micro-map, import it into related software to generate terrain, import it into Twinmotion software for rendering, and then import it into Unreal Engine to generate basic terrain;
[0016] Obtain the Shp data of building roads through the water micro-map, process it through SketchUp software to generate the surrounding 3D scene, import it into Twinmotion software for format conversion, and then import it into Unreal Engine for position adjustment;
[0017] Generate OSGB data oblique photography from aerial photos, convert the format through CisumLab to generate 3D tiles files, and import them into Unreal Engine to achieve integration with terrain;
[0018] Determine the accuracy of infrastructure modeling according to task requirements, use BIM for modeling, import into Unreal Engine and adjust the position, and then render the model material;
[0019] Vegetation and vehicles were added using Twinmotion software and imported into Unreal Engine.
[0020] As an improvement of the above method, the step 2) specifically includes:
[0021] Select the target sample in the digital twin scene;
[0022] Enable rendering of custom depth channel;
[0023] Set up custom mold channels;
[0024] Adjust the scene to a suitable position, place the camera, and adjust the viewing angle to the cutscene viewport;
[0025] Add level sequences and create cutscenes;
[0026] Export custom render sequences;
[0027] Find the corresponding custom mold value from the rendered target sample, draw the minimum circumscribed rectangle through the pixels of different mold values, and then get the bounding box of different targets. Then convert the sample set generated in the digital twin environment into a bounding box annotation to generate a virtual sample set corresponding to the target task.
[0028] As an improvement of the above method, the information matching between the boundary box of the risk potential object and the changed area based on the similarity algorithm specifically includes:
[0029] For the i-th object in the detected risk hidden object bounding box set B, the j-th bounding box area B ij and the kth change region A of the change region set A k , information fusion is performed based on the maximum SSIM structural similarity algorithm to achieve information matching between the boundary box of the risk hidden object and the changed area.
[0030] As an improvement of the above method, the method further comprises:
[0031] According to the risk objects and matching information, the change status of the risk objects is summarized, which includes addition, deletion, expansion and reduction. For the regional changes of atypical objects that are not included in the risk object boundary box, the definition and calibration of the atypical areas are further improved through manual review. If the sample is incomplete, the sample is expanded.
[0032] On the other hand, the present invention proposes a surrounding environment risk source intelligent identification system based on a virtual sample set, the system comprising:
[0033] A detection module, used to input the video image of the environment to be detected into a pre-established and trained risk hidden object detection model and an SCD change area detection model, respectively, to detect a risk hidden object boundary box set and a change area set; and
[0034] The matching and identification module is used to match the information of the boundary box of the risk potential object and the changed area based on the similarity algorithm and track the change situation, thereby realizing the identification of the risk source of the surrounding environment;
[0035] The risk hidden danger object detection model adopts Yolov5 network, and the SCD change area detection model adopts full convolutional neural network. The risk hidden danger object detection model and the SCD change area detection model are trained by constructing a digital twin scene to generate a virtual sample set.
[0036] In a third aspect, the present invention proposes a computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the computer program.
[0037] In a fourth aspect, the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes any one of the methods described above.
[0038] Compared with the prior art, the advantages of the present invention are:
[0039] 1. Use virtual sample sets to identify risks and detect regional changes in the intelligent field of surrounding environmental risk sources. The construction of virtual samples is highly flexible, adaptable and low-cost.
[0040] 2. Render the samples in layers based on Unreal Engine, set different custom depth stencil values for different samples, add camera switching according to the target task requirements, generate Sequence level sequences, and export them as training samples in custom rendering mode and select appropriate rendering channels;
[0041] 3. Find the corresponding custom mold value from the rendered virtual sample set, and draw the minimum circumscribed rectangle of the mold value according to the pixel points of different mold values to achieve automatic labeling;
[0042] 4. Through the fusion method of risk hidden danger object identification and change area detection, the simultaneous solution of object-oriented and change-oriented high-precision identification is achieved;
[0043] 5. This application is not limited to the railway field. As long as it is aimed at the construction of virtual sample sets, identification of surrounding environmental risk sources and intelligent identification of regional changes, the application will still be valid. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a flow chart of the method for intelligently identifying surrounding environment risk sources based on a virtual sample set of the present invention;
[0045] Figure 2 It is a flowchart for establishing a digital twin scenario;
[0046] Figure 3 It is the flowchart of virtual sample set generation. DETAILED DESCRIPTION
[0047] In view of the regional changes in the railway environment and the intrusion of foreign objects, the main detection method is manual collection. The types of samples collected are limited and lacking. In addition, professional image acquisition equipment is required for image data collection, and professional technicians are required to build the equipment to ensure that effective data from all angles can be collected. This leads to high image data collection costs and a long collection process. For example, the maintenance of railway infrastructure requires airspace application due to equipment limitations such as contact networks, which makes it difficult to collect images of the surrounding environment. In addition, due to manual intervention in the image annotation process, there are a large number of repetitive operations, which leads to certain deviations in the manually annotated labels. Therefore, by constructing a virtual sample set, the sample types are enriched, the number of samples is increased, the training bias problem caused by sample imbalance is solved, the model generalization ability is improved, and the method of automatic annotation of virtual samples is used to avoid errors caused by manual annotation and improve the accuracy of the training model. Based on the generated virtual sample set and automatically labeled data, Yolov5 target detection is used to train the risk hazard identification model and the fully convolutional neural network is used to train the SCD scene change detection model. The risk source categories and change areas are further mapped to simultaneously solve the object-oriented and change-oriented foreign object intrusion monitoring problems, and locate the intruding foreign objects and their changes.
[0048] This application proposes a method for intelligently identifying surrounding environment risk sources based on a virtual sample set. First, for different risk source categories, a three-dimensional virtual sample set in multiple categories and multiple scenarios is constructed based on the Unreal Engine platform. By setting a custom depth mold value and adding a level sequence, the virtual samples are layered and rendered and automatically labeled, reducing the amount of manual labeling and improving the efficiency and accuracy of labeling. Secondly, the risk potential objects are identified through Yolov5's target detection, and the change area is identified using the fully convolutional neural network scene change detection (SCD). Combining the analysis results of the two, the risk source category and the change area are mapped to achieve tracking of the change status of the risk source object (addition, disappearance, expansion, reduction), etc.
[0049] The steps include:
[0050] The video image of the environment to be detected is input into the pre-established and trained risk hidden danger object detection model and SCD change area detection model respectively, and the risk hidden danger object boundary box set and the change area set are detected respectively;
[0051] Based on the similarity algorithm, the information of the risk potential object boundary box and the change area is matched and the change situation is tracked, thereby realizing the identification of the risk source of the surrounding environment;
[0052] The risk hidden danger object detection model adopts the Yolov5 network, and the SCD change area detection model adopts a fully convolutional neural network. The risk hidden danger object detection model and the SCD change area detection model are trained based on the construction of a digital twin scene to generate a virtual sample set.
[0053] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0054] Example 1
[0055] like Figure 1 As shown, Embodiment 1 of the present invention proposes an intelligent identification method for surrounding environment risk sources based on a virtual sample set. Specifically, it includes:
[0056] 1. Creation of Virtual Scene
[0057] According to the external environment risk source category identification tasks and requirements, a digital twin scene is created based on Unreal Engine UE4, which can render sample data from different angles. The digital twin scene has the following characteristics:
[0058] (1) The rendering samples are highly realistic, ensuring the accuracy and precision of the model trained based on the virtual sample set;
[0059] (2) It can generate sample data from different scene perspectives, with fast rendering speed, and can generate a large number of multi-scene sample sets in a short time;
[0060] (3) The scenario is easy to modify, which makes it convenient to fine-tune the risk source category model according to actual conditions and has high adaptability.
[0061] According to the expected tasks and requirements, the digital twin scene corresponding to the target risk source category generated image is constructed using Unreal Engine as the platform. The scene is mainly composed of terrain, oblique photography, scene model, BIM model, vegetation, vehicles, lighting, visual effects, etc.
[0062] The terrain and image Tiff data, buildings, roads and watershed shp (Shapefile: a file format based on vector data, used to store geographic features and attribute information) data of the project are obtained through the water micro-map, and then the data is imported into relevant software (such as SuperMap, CityEngine, etc.) to generate terrain, buildings and roads, forming a digital twin large scene model.
[0063] The scenes around railways are diverse, and it is costly to obtain oblique photography models of the surrounding environment. It is impossible to obtain updated data in time according to changes in the surrounding environment, and the range of oblique photography data is limited. Considering that the macroscopic scenes of railways are not much different, terrain data can be integrated with existing oblique photography to ensure that the scene is close to reality.
[0064] Risk hazards are modeled using BIM, and then imported into the Unreal Engine through Datasmith for position adjustment. For individual components with missing materials, materials are edited and rendered to eventually form a static mesh. The surrounding terrain, surrounding buildings, roads, etc. that have been built are imported into Twinmotion for the addition and rendering of vegetation and vehicles. Flowers, grass, trees and other vegetation and vehicles in Twinmotion can be imported into the Unreal Engine losslessly. The light source in the scene uses the UltraDynamicSky dynamic skylight system for realistic effects, and then drag in visual effects Actors: volumetric clouds, post-processing volumes, exponential height fog, etc., whose main function is to enhance visualization effects. By setting up visual effects Actors with the powerful Niagara particle system, target mission scenes in different weather and seasons can be rendered, so that the virtual construction scene corresponds one-to-one with the multi-modality of the real world. The Unreal Engine digital twin scene construction technology route is as follows: Figure 2 shown.
[0065] 2. Generation of Virtual Sample Sets
[0066] In the digital twin scene, the target samples are rendered in layers and segmented with different colors to obtain the target mask image. The target object bounding box is automatically annotated based on the mask image to generate a virtual sample set corresponding to the target task, including the original constructed image and object annotated image for risk hazard identification, as well as the original constructed image, changed constructed image and mask image of the changed area for area change detection.
[0067] The virtual sample set generation process is as follows: Figure 3 shown.
[0068] The specific operations of generating a virtual sample set are as follows:
[0069] Step 1: Select the target sample in the digital twin scene and check "Custom Depth Channel"
[0070] Step 2: Set the "Custom Mold Channel" value for the target sample.
[0071] Use custom stencil for layered rendering. The custom stencil is a string of numbers in the scene. You need to modify the material of CustomStencil to remove the numbers on the screen.
[0072] Step 3: Modify the CustomStencil material
[0073] Find the CustomStencil material in the engine folder and modify the material blueprint:
[0074] The material domain is changed to "post-processing", the scene texture function is called, the scene texture ID is changed to "custom mold", and the "color" pin calls the "mask" function to mask the scene texture. At this time, the output value is divided into two types: one is divided by 255, and the value obtained at this time is the mold value 0~255, and a random color is used to assign color to each mold value. The other is to set the custom depth channel to 0. Multiply the two results and connect them to the material's self-illumination to get the corresponding effect.
[0075] The settings of custom depth mold values for simultaneous multi-sample detection are shown in the table
[0076] Table 1 Example of custom mold value settings
[0077] Target sample name Custom Depth Stencil Values RGB Value Dangerous Tree 1 213,166,56 Light floating objects 2 105,184,128 Telephone pole 3 102,203,210 signal tower 4 120,47,180 Anti-throwing net 5 113,134,18 vehicle 6 222,37,118 figure 7 154,40,155 Color steel tile house 8 40,71,207 Fence 9 26,196,35 Dust screen 10 151,175,89 the remaining 0 0,0,0 ……
[0078] Step 4: Add a camera
[0079] Adjust the scene to a suitable position, place the camera, and adjust the viewing angle to the cutscene viewport to check whether the generated sample set meets the task requirements.
[0080] Step 5: Add level sequences and create cutscenes
[0081] In the cutscene, camera switching is added at the appropriate time node to generate the cutscene. The purpose is to construct virtual samples of the same risk source in different spatial locations, expand the breadth of the virtual sample set, and be able to fit various distribution modes of actual scenes.
[0082] Step 6: Export a custom render sequence
[0083] When exporting, you need to change the image output format to "Custom Rendering Channel", set the frame rate and resolution that match the project requirements, and add a rendering channel. Custom rendering channels include: scene depth, custom mold, final imaging and other formats. According to the project requirements, select "Custom Mold" and "Final Imaging" here.
[0084] Step 7: Automatic labeling
[0085] The corresponding custom mold value is found from the rendered virtual sample set, and the minimum circumscribed rectangle is drawn through the pixel points of different mold values to obtain the bounding box of different targets. The sample set generated in the digital twin environment is then converted into a bounding box annotation. The automatic annotation of training data is realized in the process, which reduces the tedious work of manually annotating pictures and improves work efficiency and accuracy.
[0086] Step 8: After the above operations, a virtual sample set library under various risk source categories and multiple scenarios is obtained.
[0087] 3. Intelligent Identification
[0088] 1. Construct Yolov5 risk identification model
[0089] Based on the constructed virtual sample set, including the original constructed image containing hidden danger objects and the annotated image with automatically annotated object bounding boxes, where the object type set is [t1, t2, t3, ..., tm], with a total of m object types, the Yolov5 risk hidden danger target detection model is trained to realize the detection of risk hidden danger objects.
[0090] 2. Build SCD scene change detection model
[0091] Based on the constructed virtual sample set, including the original constructed image before the scene change, the changed constructed image after the scene change, and the mask image of the changed area, the SCD scene change detection model based on the fully convolutional neural network is trained to identify the changed area and achieve accurate positioning and segmentation of the changed area.
[0092] 3. Object area matching and change tracking based on similarity algorithm
[0093] For the captured video images, the optimal Yolov5 risk hidden object detection model obtained through training is used to detect the risk hidden object set T, for example, T = [t 11 (the first t1 object), t 12 (the second t1 object), t 21 (the first t2 object), t 31 (the first t3 object)], and its corresponding bounding box set B, such as B = [B 11 (bounding box of the first t1 object), B 12 (the bounding box of the second t1 object), B 21 (bounding box of the first t2 object), B 31(the bounding box of the first t3 object)]; the optimal SCD change region detection model obtained through training is used to identify the change region A, for example, A = [A1, A2, A3, A4, A5]. The detected risk hidden object bounding box set B is compared with the identified change region set A, and the detected bounding box region B is compared. ij (i is the i-th object, j is the j-th bounding box) and the detected change area A k (k is the kth change region) Based on the maximum SSIM structural similarity algorithm (max(SSIM(A k ,B ij )))Fuse the bounding box and the region information to achieve information matching between the object bounding box and the changed region.
[0094] Furthermore, according to the object detection results and the matching information between the object and the area, the change status of the object is summarized, such as addition, deletion, expansion, and reduction. For the regional changes of atypical objects, the changed areas are located, and the definition and calibration of atypical areas are further improved through manual review, and the sample coding library is expanded. Finally, through the object area fusion technology, the changes of risk objects are fully grasped, and the regional change trend tracking of atypical objects and the expansion of the sample coding library are solved simultaneously.
[0095] In summary, based on the construction of multi-category, multi-modal virtual sample sets and automatically labeled data that meet the task requirements, the risk hazard identification model is trained through Yolov5 target detection and the SCD scene change detection model is trained through the full convolutional neural network. The risk source category and the change area are further mapped, and the object-oriented and change-oriented foreign body intrusion monitoring problems are solved simultaneously, and the intrusion foreign bodies and their changes are located. It lays a technical foundation for the intelligent identification and situation tracking of risk sources in the surrounding environment of railways, and provides efficient protection for the safety of the surrounding areas of high-speed railway operations.
[0096] Example 2
[0097] Embodiment 2 of the present invention proposes a system for intelligently identifying surrounding environment risk sources based on a virtual sample set, the system comprising:
[0098] A detection module is used to input the video image of the environment to be detected into the pre-established and trained risk hidden danger object detection model and SCD change area detection model, respectively, to detect the risk hidden danger object boundary box set and the change area set;
[0099] The matching and identification module is used to match the information of the boundary box of the risk potential object and the changed area based on the similarity algorithm and track the change situation, thereby realizing the identification of the risk source of the surrounding environment;
[0100] The risk hidden danger object detection model adopts the Yolov5 network, and the SCD change area detection model adopts a fully convolutional neural network. The risk hidden danger object detection model and the SCD change area detection model are trained based on the construction of a digital twin scene to generate a virtual sample set.
[0101] An embodiment of the present invention may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It can be understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.
[0102] The user interface may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).
[0103] It is understood that the memory in the disclosed embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0104] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.
[0105] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.
[0106] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:
[0107] Execute the steps of the method of Example 1.
[0108] The method of embodiment 1 can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in embodiment 1 can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with embodiment 1 can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0109] It is understood that the embodiments described in the present invention can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.
[0110] For software implementation, the technology of the present invention can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present invention. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0111] The embodiment of the present invention may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.
Claims
1. A method for intelligently identifying surrounding environment risk sources based on a virtual sample set, the method comprising: The video image of the environment to be detected is input into the pre-established and trained risk hidden danger object detection model and SCD change area detection model respectively, and the risk hidden danger object boundary box set and the change area set are detected respectively; Based on the similarity algorithm, the information of the risk potential object boundary box and the change area is matched and the change situation is tracked, thereby realizing the identification of the risk source of the surrounding environment; The risk hidden danger object detection model adopts the Yolov5 network, and the SCD change area detection model adopts the fully convolutional neural network. The risk hidden danger object detection model and the SCD change area detection model are trained by generating a virtual sample set by constructing a digital twin scene.
2. The method for intelligently identifying surrounding environment risk sources based on a virtual sample set according to claim 1 is characterized in that: The generation of a virtual sample set by constructing a digital twin scene specifically includes: Step 1) Create a digital twin scene based on Unreal Engine UE4; Step 2) Render the target samples in layers in the digital twin scene, segment them with different colors, obtain the target mask image, automatically annotate the target object bounding box based on the target mask image, and generate a virtual sample set corresponding to the target task.
3. The method for intelligently identifying surrounding environment risk sources based on a virtual sample set according to claim 1 or 2, characterized in that: The digital twin scene in step 1) includes: terrain, oblique photography, scene model, BIM model, vegetation, vehicles, lighting and visual effects.
4. The method for intelligently identifying surrounding environment risk sources based on a virtual sample set according to claim 3 is characterized in that: The step 1) specifically includes: Obtain terrain image data through the water micro-map, import it into related software to generate terrain, import it into Twinmotion software for rendering, and then import it into Unreal Engine to generate basic terrain; Obtain the Shp data of building roads through the water micro-map, process it through SketchUp software to generate the surrounding 3D scene, import it into Twinmotion software for format conversion, and then import it into Unreal Engine for position adjustment; Generate OSGB data oblique photography from aerial photos, convert the format through CisumLab to generate 3D tiles files, and import them into Unreal Engine to achieve integration with terrain; Determine the accuracy of infrastructure modeling according to task requirements, use BIM for modeling, import into Unreal Engine and adjust the position, and then render the model material; Vegetation and vehicles were added using Twinmotion software and imported into Unreal Engine.
5. The method for intelligently identifying surrounding environment risk sources based on virtual sample sets according to claim 2 is characterized in that: The step 2) specifically includes: Select the target sample in the digital twin scene; Enable rendering of custom depth channel; Set up a custom mold channel; adjust the scene to a suitable position, place the camera, and adjust the viewing angle to the cutscene viewport; Add level sequences and create cutscenes; Export custom render sequences; Find the corresponding custom mold value from the rendered target sample, draw the minimum enclosing rectangle through the pixel points of different mold values, obtain the bounding box of different targets, and then convert the sample set generated in the digital twin environment into a bounding box annotation to generate a virtual sample set corresponding to the target task.
6. The method for intelligently identifying surrounding environment risk sources based on a virtual sample set according to claim 1, characterized in that: The information matching between the boundary box of the risk potential object and the changed area based on the similarity algorithm specifically includes: For the i-th object in the detected risk hidden object bounding box set B, the j-th bounding box area B ij and the kth change region A of the change region set A k , information fusion is performed based on the maximum SSIM structural similarity algorithm to achieve information matching between the boundary box of the risk hidden object and the changed area.
7. The method for intelligently identifying surrounding environment risk sources based on a virtual sample set according to claim 1, characterized in that: The method further comprises: Summarize the change status of the risk hidden danger object according to the matching information between the boundary box of the risk hidden danger object and the change area, wherein the change status includes addition, deletion, expansion and reduction; For regional changes of atypical objects that are not included in the boundary box of risk objects, the definition and calibration of atypical areas are further improved through manual review. If the sample is incomplete, the sample is expanded.
8. An intelligent identification system for surrounding environment risk sources based on a virtual sample set, characterized in that: The system comprises: A detection module, used to input the video image of the environment to be detected into a pre-established and trained risk hidden danger object detection model and an SCD change area detection model, respectively, to detect a risk hidden danger object boundary box set and a change area set; and The matching and identification module is used to match the information of the boundary box of the risk potential object and the changed area based on the similarity algorithm and track the change situation, thereby realizing the identification of the risk source of the surrounding environment; The risk hidden danger object detection model adopts the Yolov5 network, and the SCD change area detection model adopts a fully convolutional neural network to train the risk hidden danger object detection model and the SCD change area detection model by constructing a digital twin scene to generate a virtual sample set.
9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 8.