AR video superposition river and lake treatment system and method based on machine vision recognition

Abstract

The invention belongs to the technical field of artificial intelligence and AR, and discloses an AR video superposition river and lake treatment system and method based on machine vision recognition, and the method comprises the steps: a data collection module collects perception data and video stream data, and a machine vision processing module transmits the video stream data to a server; a machine vision algorithm is used for carrying out automatic analysis and processing, pollutants and abnormal behaviors are identified, an augmented reality display module superposes the identified pollutants, the abnormal behaviors and perception data into a real-time image of monitoring equipment through an AR technology, a real scene is fed back, visual display is provided, and an early warning and decision support module carries out early warning according to a processing result. An early warning mechanism is automatically triggered, related personnel are notified through multiple modes, treatment decision support is provided at the same time, by means of the mode, the river and lake treatment system is fused with the AR and machine vision technology, expensive AR equipment is not needed, and existing perception data and video data are fully activated and efficiently utilized.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and AR technology, and in particular to a system and method for river and lake management based on machine vision recognition and AR video overlay. Background Art

[0002] Augmented Reality (AR) is a technology that overlays virtual information (including text, images, videos, and 3D models) on the real world in real time. Machine vision recognition is a key branch of artificial intelligence. It uses computers and corresponding hardware to process and analyze images and videos to automatically identify and understand objects, scenes, actions, and other content.

[0003] In existing technical scenarios, the combination of the two technologies often requires the use of AR headsets to collect video images. For the scenarios of river and lake water environment management, personal wearable AR devices are not suitable for command, decision-making and operation and maintenance needs, and the cost of AR headsets is relatively high. At the same time, in river and lake water environment management projects, sensing devices and video surveillance systems have been widely deployed, and these sensing data and video data have not been fully activated and efficiently utilized.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide an AR video overlay river and lake management system and method based on machine vision recognition, aiming to solve the technical problems that when combining machine vision and AR technology, the cost of AR equipment is high and the existing perception data and video data are not fully activated and efficiently utilized.

[0006] To achieve the above objectives, the present invention provides an AR video superimposed river and lake management system based on machine vision recognition, which includes: a data acquisition module, a machine vision processing module, an augmented reality display module, and an early warning and decision support module; The data acquisition module is used to collect perception data through the perception device and collect video stream data through the monitoring device, and upload the perception data and the video stream data to the server; The machine vision processing module is used to forward the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, extract key features from the preprocessed video image, and use a neural network model trained by deep learning to identify pollutants and abnormal behavior in the video image based on the key features; The augmented reality display module is configured to obtain environmental data based on the perception data, obtain feature points of the surrounding environment using a visual sensor based on the environmental data, construct an environmental map corresponding to the video image based on the feature points, obtain actual coordinates of the pollutants and abnormal behavior in the environmental map using coordinate transformation, construct virtual information and render the virtual information, and overlay the rendered virtual information onto the video image based on the actual coordinates; The early warning and decision support module is used to automatically trigger an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notify relevant personnel to provide governance decision support.

[0007] In some embodiments, the machine vision processing module is used to perform denoising on the video image using image denoising technology, and at the same time adjust the brightness and contrast of the video image to obtain a preprocessed video image, identify and extract image features of the preprocessed video image, and use feature description to convert the image features into mathematical representations to obtain key features.

[0008] In some embodiments, the augmented reality display module is used to obtain the world coordinates of the pollutants and abnormal behaviors based on the world coordinate system, convert the world coordinates into camera coordinates based on the camera coordinate system, convert the camera coordinates into image coordinates based on the image coordinate system, and convert the image coordinates into pixel coordinates based on the pixel coordinate system, so as to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map.

[0009] In some embodiments, the world coordinate system is a coordinate system with reference to the actual position of the visual sensor, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of the point in the image coordinate system, and the pixel coordinates describe the arrangement of pixels in the image.

[0010] In some embodiments, the augmented reality display module is used to generate virtual information according to actual needs, and the virtual information includes three-dimensional models, animations or text information. The virtual information is edited, and the Unity3D rendering engine is selected to render the edited virtual information in real time. The rendered virtual information is superimposed on the video image based on the actual coordinates, and the rendering parameters are adjusted in real time during the superposition process.

[0011] In some embodiments, the early warning and decision support module is used to clean the perception data and the video image processing results of the augmented reality display module, analyze the cleaned data to identify water quality change trends and potential risks, evaluate the comprehensive early warning level of rivers and lakes and the early warning level of individual indicators based on the water quality change trends and potential risks, push the comprehensive early warning level and the early warning level of the individual indicators to the augmented reality display module, and notify relevant personnel in various ways to provide governance decision support.

[0012] In addition, to achieve the above-mentioned purpose, the present invention also proposes a method for river and lake management based on machine vision recognition using AR video superimposed on the image. The method for river and lake management based on machine vision recognition uses the AR video superimposed on the image river and lake management system based on machine vision recognition as described above. The method for river and lake management based on AR video superimposed on the image river and lake management system includes: The data acquisition module collects sensory data through the sensory device and collects video stream data through the monitoring device, and uploads the sensory data and the video stream data to the server; The machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, extract key features from the preprocessed video image, and use a neural network model trained by deep learning to identify pollutants and abnormal behavior in the video image based on the key features; The augmented reality display module obtains environmental data based on the perception data, obtains feature points of the surrounding environment through a visual sensor based on the environmental data, constructs an environmental map corresponding to the video image based on the feature points, obtains actual coordinates of the pollutants and abnormal behaviors in the environmental map through coordinate transformation, constructs virtual information and renders the virtual information, and superimposes the rendered virtual information on the video image based on the actual coordinates; The early warning and decision support module automatically triggers an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide governance decision support.

[0013] In some embodiments, the machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, and extract key features from the preprocessed video image, including: The machine vision processing module uses image denoising technology to denoise the video image, and adjusts the brightness and contrast of the video image to obtain a preprocessed video image, identifies and extracts image features of the preprocessed video image, and uses feature description to convert the image features into mathematical representations to obtain key features.

[0014] In some embodiments, the obtaining the actual coordinates of the pollutants and abnormal behaviors in the environment map by using coordinate transformation includes: The augmented reality display module obtains the world coordinates of the pollutants and abnormal behaviors based on the world coordinate system, converts the world coordinates into camera coordinates based on the camera coordinate system, converts the camera coordinates into image coordinates based on the image coordinate system, and converts the image coordinates into pixel coordinates based on the pixel coordinate system to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map.

[0015] In some embodiments, the world coordinate system is a coordinate system with reference to the actual position of the visual sensor, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of the point in the image coordinate system, and the pixel coordinates describe the arrangement of pixels in the image.

[0016] The data acquisition module in the present invention collects sensory data through sensory devices and video stream data through monitoring devices, and uploads the sensory data and the video stream data to a server; the machine vision processing module forwards the video stream data to the AI ​​recognition server to pre-process the video image corresponding to the video stream data, extract key features from the pre-processed video image, and use a neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features; the augmented reality display module obtains environmental data based on the sensory data, obtains feature points of the surrounding environment through a visual sensor according to the environmental data, and uses the key features to identify pollutants and abnormal behaviors in the video image based on the key features. The environmental map corresponding to the video image is constructed by collecting points, the actual coordinates of the pollutants and abnormal behaviors in the environmental map are obtained by using coordinate transformation, virtual information is constructed and rendered, and the rendered virtual information is superimposed on the video image based on the actual coordinates; the early warning and decision support module automatically triggers the early warning mechanism according to the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide management decision support. Through the above method, the river and lake management system is integrated with AR and machine vision technology, without the need for expensive AR equipment, and the existing perception data and video data are fully activated and efficiently utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1This is a structural block diagram of the first embodiment of the AR video overlay river and lake management system based on machine vision recognition of the present invention; Figure 2 This is a schematic diagram of the spatial positioning process in the AR video superimposed river and lake management system based on machine vision recognition of the present invention; Figure 3 This is a schematic diagram of the coordinate system conversion relationship in the AR video superimposed river and lake management system based on machine vision recognition of the present invention; Figure 4 This is a flow chart of the first embodiment of the method for river and lake management using AR video overlay based on machine vision recognition according to the present invention.

[0018] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] The embodiment of the present invention provides an AR video superimposed river and lake management system based on machine vision recognition, referring to Figure 1 , Figure 1 This is a structural block diagram of the first embodiment of the AR video overlay river and lake management system based on machine vision recognition in the present invention.

[0021] In this embodiment, the AR video overlay river and lake management system based on machine vision recognition includes: a data acquisition module 10, a machine vision processing module 20, an augmented reality display module 30, and an early warning and decision support module 40. Among them, the data acquisition module 10 uses front-end equipment such as video monitoring equipment, water level meters, and water quality meters to collect data on river and lake water levels, water quality, water surface anomalies, shorelines, and surrounding environments in real time. The machine vision processing module 20 transmits the collected videos and images to the server, and uses machine vision algorithms to automatically analyze and process them to identify pollutants and abnormal behavior. The augmented reality display module 30 uses AR technology to overlay the identified pollutants, abnormal behavior, and perception data onto the real-time image of the monitoring equipment, feedback the real scene, and provide intuitive display. The early warning and decision support module 40 automatically triggers the early warning mechanism based on the processing results, notifies relevant personnel through various means, and provides management decision support.

[0022] In a specific implementation, the data acquisition module 10 collects sensory data through sensory devices and video stream data through monitoring devices, and uploads the sensory data and video stream data to a server. The sensory devices include water level meters and water quality meters. The data acquisition module 10 transmits the sensory data to the server using a wireless transmission method based on GPRS, CDMA, 3G / 4G, etc. with the help of a remote terminal unit. During the transmission process, protocols such as Modbus, Profibus, and EtherNet / IP are followed. To ensure data security, encryption algorithms including AES and RSA are supported. For video monitoring equipment, it is connected to the streaming media server through protocols such as GAT1400, GB28181, RTSP, Onvif, SDK, and RTMP to complete the collection and storage of video streams.

[0023] The machine vision processing module 20 forwards the video stream data to the AI ​​recognition server to pre-process the video image corresponding to the video stream data, extract key features from the pre-processed video image, and use the neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features. In this embodiment, image pre-processing technology is used to perform pre-processing operations such as denoising, contrast enhancement, and filtering on the collected image to improve image quality and accuracy. Specifically, the machine vision processing module 20 uses image denoising technology to denoise the video image, and at the same time adjusts the brightness and contrast of the video image to obtain a pre-processed video image, identifies and extracts the image features of the pre-processed video image, and uses feature description to convert the image features into mathematical representations to obtain key features. It should be noted that denoising technology, such as applying median filtering and Gaussian filtering, can make the image clearer through brightness and contrast, and identify and extract features such as edges and textures in the image. After extracting key features, a neural network model trained through deep learning is used to identify pollutants and abnormal behavior in the video images based on these key features. Specifically, deep learning techniques are used to train the neural network model, automatically learning complex patterns in the images. This process includes: 1. Creating a training dataset. 2. Applying the trained neural network model. 3. Identifying the object to be detected and comparing it with the model.

[0024] Furthermore, the augmented reality display module 30 obtains environmental data based on the perception data, obtains feature points of the surrounding environment through a visual sensor according to the environmental data, constructs an environmental map corresponding to the video image based on the feature points, obtains the actual coordinates of the pollutants and abnormal behaviors in the environmental map using coordinate transformation, constructs virtual information and renders the virtual information, and superimposes the rendered virtual information on the video image based on the actual coordinates.

[0025] It should be noted that in this embodiment, the scene in the video can be spatially positioned by SLAM (Simultaneous Localization and Mapping) technology to determine the precise location where the virtual information should be superimposed. After obtaining the feature points, the position and angle of the visual sensor are used to construct a map of the environment. The above spatial positioning process can be referred to Figure 2 As shown, data acquisition involves acquiring environmental data, visual odometry involves acquiring feature points of the surrounding environment through visual sensors, back-end optimization involves improving positioning and map construction accuracy through optimization algorithms, mapping involves constructing a map of the environment based on the position and angle of the visual sensors, and target recognition involves using deep learning technology to train a neural network model to automatically identify target objects in the image, outline the object contours, and convert coordinates to obtain the actual coordinates of the identified object based on the conversion relationship between world coordinates and camera coordinates. The specific process of coordinate conversion is to obtain the world coordinates of the pollutant and abnormal behavior based on the world coordinate system, convert the world coordinates to camera coordinates based on the camera coordinate system, convert the camera coordinates to image coordinates based on the image coordinate system, and convert the image coordinates to pixel coordinates based on the pixel coordinate system to obtain the actual coordinates of the pollutant and abnormal behavior in the environmental map.

[0026] It should be noted that this embodiment involves four coordinate systems, namely the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The world coordinate system is a coordinate system based on the actual position of the visual sensor, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of the point in the image coordinate system, and the pixel coordinate system describes the arrangement of pixels in the image. For specific conversion relationships, please refer to Figure 3 As shown, Figure 3 The world coordinate system shown in is converted to the camera coordinate system - the camera coordinate system is calculated through the transformation of the world coordinate system (rotation R, translation T). It is the process of converting the point coordinates P (Xw, Yw, Zw) described in the world coordinate system into P (Xc, Yc, Zc) described in the camera coordinate system; the camera coordinate system is converted to the image coordinate system - relying on the camera's intrinsic parameter f, the P (Xc, Yc, Zc) described in the camera coordinate system is converted into image coordinates, and the similarity of the graphics is used to achieve 3D to 2D conversion; the image coordinate system is converted to the pixel coordinate system - the pixel coordinate system and the image coordinate system are both on the imaging plane. The correspondence between the unit mm of the image coordinate system and the unit pixel of the pixel coordinate system is defined to achieve mutual conversion.

[0027] In this embodiment, the augmented reality display module 30 constructs virtual information and renders the virtual information. The process of superimposing the rendered virtual information on the video image based on the actual coordinates is specifically to generate virtual information according to actual needs, the virtual information includes three-dimensional models, animations or text information, edit the virtual information, select the Unity3D rendering engine to render the edited virtual information in real time, superimpose the rendered virtual information on the video image based on the actual coordinates, and adjust the rendering parameters in real time during the superposition process.

[0028] It should be noted that the virtual information constructed by three-dimensional modeling has a high degree of realism and interactivity. The editing of virtual information includes the adjustment of attributes such as color, transparency, and size, so as to ensure its harmonious integration with the real scene. Real-time adjustment of rendering parameters during the superposition process can make the virtual information and the real scene perfectly integrated. The adjustment method can be selected according to actual needs, and this is not limited in this embodiment. Finally, this embodiment can also interact and provide feedback with the user, such as user interaction: providing the user with an interactive interface so that it can interact with the virtual information. Feedback mechanism: based on the user's interactive behavior, the status or presentation method of the virtual information is updated in real time to improve the user experience.

[0029] In this embodiment, an early warning and decision support module 40 is also provided. The early warning and decision support module 40 automatically triggers an early warning mechanism according to the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide governance decision support. Specifically, the perception data and the video image processing results of the augmented reality display module are cleaned, and the cleaned data is analyzed to identify water quality change trends and potential risks. The comprehensive early warning level of rivers and lakes and the early warning level of individual indicators are evaluated based on the water quality change trends and potential risks, and the comprehensive early warning level and the early warning level of the individual indicators are pushed to the augmented reality display module, and relevant personnel are notified in various ways to provide governance decision support.

[0030] It should be noted that the data cleaning in this embodiment includes but is not limited to removing outliers, missing values, etc. Data cleaning can ensure data quality, and the identification of water quality change trends and potential risks can be achieved by using statistics and data mining techniques to process the cleaned data. The determination of the warning level is divided into single-factor warning and comprehensive warning. The single-factor warning is specifically based on the surface water environmental quality standard, comparing each water quality indicator with the corresponding level of warning limit value, so as to determine the warning level of different single indicators. The comprehensive warning is to combine the gray clustering model and other methods to comprehensively consider multiple water quality indicators, so as to comprehensively evaluate the comprehensive warning level of river and lake water quality. The comprehensive warning can, for example, weight multiple single water quality indicators to obtain a comprehensive water quality indicator, and then compare the comprehensive water quality indicator with the warning limit value to obtain a comprehensive warning level. The specific method and the set warning limit value can be adjusted based on actual needs. This is not limited in this embodiment. The above method is only used for illustration.

[0031] In this embodiment, the data acquisition module collects sensory data through the sensory device and collects video stream data through the monitoring device, and uploads the sensory data and the video stream data to the server; the machine vision processing module forwards the video stream data to the AI ​​recognition server to pre-process the video image corresponding to the video stream data, extract key features from the pre-processed video image, and use the neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features; the augmented reality display module obtains environmental data based on the sensory data, obtains feature points of the surrounding environment through the visual sensor according to the environmental data, and The feature points construct an environmental map corresponding to the video image, and the actual coordinates of the pollutants and abnormal behaviors in the environmental map are obtained by coordinate transformation. Virtual information is constructed and rendered, and the rendered virtual information is superimposed on the video image based on the actual coordinates. The early warning and decision support module automatically triggers the early warning mechanism according to the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide management decision support. Through the above method, the river and lake management system is integrated with AR and machine vision technology, without the need for expensive AR equipment, and the existing perception data and video data are fully activated and efficiently utilized.

[0032] Reference Figure 4 , Figure 4 This is a flow chart of the first embodiment of the method for river and lake management using AR video overlay based on machine vision recognition according to the present invention.

[0033] like Figure 4 As shown, the method for river and lake management based on AR video superimposition based on machine vision recognition proposed in an embodiment of the present invention includes: Step S10: The data acquisition module collects perception data through the perception device and collects video stream data through the monitoring device, and uploads the perception data and the video stream data to the server.

[0034] Step S20: The machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, extract key features from the preprocessed video image, and use the neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features.

[0035] Step S30: The augmented reality display module obtains environmental data based on the perception data, obtains feature points of the surrounding environment through a visual sensor according to the environmental data, constructs an environmental map corresponding to the video image based on the feature points, obtains the actual coordinates of the pollutants and abnormal behaviors in the environmental map through coordinate transformation, constructs virtual information and renders the virtual information, and superimposes the rendered virtual information on the video image based on the actual coordinates.

[0036] Step S40: The early warning and decision support module automatically triggers an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide governance decision support.

[0037] In this embodiment, the data acquisition module collects sensory data through the sensory device and collects video stream data through the monitoring device, and uploads the sensory data and the video stream data to the server; the machine vision processing module forwards the video stream data to the AI ​​recognition server to pre-process the video image corresponding to the video stream data, extract key features from the pre-processed video image, and use the neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features; the augmented reality display module obtains environmental data based on the sensory data, obtains feature points of the surrounding environment through the visual sensor according to the environmental data, and The feature points construct an environmental map corresponding to the video image, and the actual coordinates of the pollutants and abnormal behaviors in the environmental map are obtained by coordinate transformation. Virtual information is constructed and rendered, and the rendered virtual information is superimposed on the video image based on the actual coordinates. The early warning and decision support module automatically triggers the early warning mechanism according to the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide management decision support. Through the above method, the river and lake management system is integrated with AR and machine vision technology, without the need for expensive AR equipment, and the existing perception data and video data are fully activated and efficiently utilized.

[0038] In some embodiments, the machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data and extract key features from the preprocessed video image, including: the machine vision processing module uses image denoising technology to denoise the video image, and at the same time adjusts the brightness and contrast of the video image to obtain the preprocessed video image, identifies and extracts image features of the preprocessed video image, and uses feature description to convert the image features into mathematical representations to obtain key features.

[0039] In some embodiments, the use of coordinate transformation to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map includes: the augmented reality display module obtains the world coordinates of the pollutants and abnormal behaviors based on the world coordinate system, converts the world coordinates into camera coordinates based on the camera coordinate system, converts the camera coordinates into image coordinates based on the image coordinate system, and converts the image coordinates into pixel coordinates based on the pixel coordinate system, so as to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map.

[0040] In some embodiments, the world coordinate system is a coordinate system with reference to the actual position of the visual sensor, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of the point in the image coordinate system, and the pixel coordinates describe the arrangement of pixels in the image.

[0041] In some embodiments, the constructing of virtual information and rendering of the virtual information, and superimposing the rendered virtual information on the video image based on the actual coordinates, include: the augmented reality display module generates virtual information according to actual needs, the virtual information includes a three-dimensional model, animation or text information, editing the virtual information, selecting the Unity3D rendering engine to render the edited virtual information in real time, superimposing the rendered virtual information on the video image based on the actual coordinates, and adjusting the rendering parameters in real time during the superposition process.

[0042] In some embodiments, the early warning and decision support module automatically triggers an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide governance decision support, including: the early warning and decision support module performs data cleaning on the perception data and the video image processing results of the augmented reality display module, analyzes the cleaned data to identify water quality change trends and potential risks, evaluates the comprehensive early warning level of rivers and lakes and the early warning level of individual indicators based on the water quality change trends and potential risks, pushes the comprehensive early warning level and the early warning level of the individual indicators to the augmented reality display module, and notifies relevant personnel in various ways to provide governance decision support.

[0043] An embodiment of the present application also provides an AR video overlay river and lake management device based on machine vision recognition, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the memory is used to store the AR video overlay river and lake management program based on machine vision recognition; the processor is used to implement the above-mentioned AR video overlay river and lake management method based on machine vision recognition when executing the program stored in the memory.

[0044] The communication bus mentioned in the aforementioned machine vision-based AR video overlay river and lake management equipment can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, and a control bus.

[0045] The communication interface is used for communication between the above-mentioned AR video overlay river and lake management equipment based on machine vision recognition and other equipment.

[0046] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Alternatively, the memory may be at least one storage device located remotely from the processor.

[0047] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0048] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0049] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0050] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0052] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any limitation on this.

[0053] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the embodiment scheme, and no limitation is made here.

[0054] In addition, for technical details not fully described in this embodiment, please refer to the AR video superimposed river and lake management method based on machine vision recognition provided in any embodiment of the present invention, which will not be repeated here.

[0055] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0056] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0057] Through the above description of the embodiments, those skilled in the art will clearly understand that the methods of the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is the more preferred implementation method. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for enabling a terminal device (such as a mobile phone, computer, server, or network device) to execute the methods described in the various embodiments of the present invention.

[0058] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0059] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

Claims

1. An AR video superimposed river and lake management system based on machine vision recognition, characterized in that: The AR video superimposed river and lake management system based on machine vision recognition includes: a data acquisition module, a machine vision processing module, an augmented reality display module, and an early warning and decision support module; The data acquisition module is used to collect perception data through a perception device and to collect video stream data through a monitoring device, and to upload the perception data and the video stream data to a server; The machine vision processing module is used to forward the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, extract key features from the preprocessed video image, and use a neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features; The augmented reality display module is used to obtain environmental data based on the perception data, obtain feature points of the surrounding environment through a visual sensor according to the environmental data, construct an environmental map corresponding to the video image based on the feature points, obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map by coordinate conversion, construct virtual information and render the virtual information, and superimpose the rendered virtual information on the video image based on the actual coordinates; The early warning and decision support module is used to automatically trigger an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notify relevant personnel to provide governance decision support.

2. The AR video superimposed river and lake management system based on machine vision recognition as claimed in claim 1, characterized in that: The machine vision processing module is used to perform denoising on the video image using image denoising technology, and adjust the brightness and contrast of the video image to obtain a preprocessed video image, identify and extract image features of the preprocessed video image, and convert the image features into mathematical representations using feature descriptions to obtain key features.

3. The AR video superimposed river and lake management system based on machine vision recognition as claimed in claim 1, characterized in that: The augmented reality display module is used to obtain the world coordinates of the pollutants and abnormal behaviors based on the world coordinate system, convert the world coordinates into camera coordinates based on the camera coordinate system, convert the camera coordinates into image coordinates based on the image coordinate system, and convert the image coordinates into pixel coordinates based on the pixel coordinate system, so as to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map.

4. The AR video superimposed river and lake management system based on machine vision recognition as claimed in claim 3 is characterized in that: The world coordinate system is a coordinate system that uses the actual position of the visual sensor as a reference, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of a point in the image coordinate system, and the pixel coordinate system describes the arrangement of pixels in the image.

5. The AR video superimposed river and lake management system based on machine vision recognition as claimed in claim 1, characterized in that: The augmented reality display module is used to generate virtual information according to actual needs, and the virtual information includes three-dimensional models, animations or text information. The virtual information is edited, and the Unity3D rendering engine is selected to render the edited virtual information in real time. The rendered virtual information is superimposed on the video image based on the actual coordinates, and the rendering parameters are adjusted in real time during the superposition process.

6. The AR video superimposed river and lake management system based on machine vision recognition as described in any one of claims 1 to 5, characterized in that: The early warning and decision support module is used to clean the perception data and the video image processing results of the augmented reality display module, analyze the cleaned data to identify water quality change trends and potential risks, evaluate the comprehensive early warning level of rivers and lakes and the early warning level of single indicators based on the water quality change trends and potential risks, push the comprehensive early warning level and the early warning level of the single indicator to the augmented reality display module, and notify relevant personnel in various ways to provide governance decision support.

7. A method for river and lake management based on AR video superposition based on machine vision recognition, characterized in that: The method for superimposing rivers and lakes with AR video based on machine vision recognition is applied to the system for superimposing rivers and lakes with AR video based on machine vision recognition as described in any one of claims 1 to 6, and the method for superimposing rivers and lakes with AR video based on machine vision recognition includes: The data acquisition module collects perception data through the perception device and collects video stream data through the monitoring device, and uploads the perception data and the video stream data to the server; The machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, extract key features from the preprocessed video image, and use a neural network model trained by deep learning to identify pollutants and abnormal behaviors in the video image based on the key features; The augmented reality display module obtains environmental data based on the perception data, obtains feature points of the surrounding environment through a visual sensor according to the environmental data, constructs an environmental map corresponding to the video image based on the feature points, obtains actual coordinates of the pollutants and abnormal behaviors in the environmental map by using coordinate transformation, constructs virtual information and renders the virtual information, and superimposes the rendered virtual information on the video image based on the actual coordinates; The early warning and decision support module automatically triggers an early warning mechanism based on the perception data and the video image processing results of the augmented reality display module, and notifies relevant personnel to provide governance decision support.

8. The method for river and lake management based on AR video superposition based on machine vision recognition as claimed in claim 7, characterized in that: The machine vision processing module forwards the video stream data to the AI ​​recognition server to preprocess the video image corresponding to the video stream data, and extract key features from the preprocessed video image, including: The machine vision processing module uses image denoising technology to denoise the video image, and adjusts the brightness and contrast of the video image to obtain a preprocessed video image, identifies and extracts image features of the preprocessed video image, and uses feature description to convert the image features into mathematical representations to obtain key features.

9. The method for river and lake management based on AR video superposition based on machine vision recognition as claimed in claim 7, characterized in that: The method of obtaining the actual coordinates of the pollutants and abnormal behaviors in the environment map by using coordinate transformation includes: The augmented reality display module obtains the world coordinates of the pollutants and abnormal behaviors based on the world coordinate system, converts the world coordinates into camera coordinates based on the camera coordinate system, converts the camera coordinates into image coordinates based on the image coordinate system, and converts the image coordinates into pixel coordinates based on the pixel coordinate system, so as to obtain the actual coordinates of the pollutants and abnormal behaviors in the environmental map.

10. The method for river and lake management based on machine vision recognition and AR video superposition as claimed in claim 9, characterized in that: The world coordinate system is a coordinate system that uses the actual position of the visual sensor as a reference, the camera coordinate system is a coordinate system of the visual sensor itself, the image coordinate system describes the imaging point position of a point in the image coordinate system, and the pixel coordinate system describes the arrangement of pixels in the image.

Images