Offset detection method, system and device and storage medium

Abstract

The invention discloses an offset detection method, system and device and a storage medium, a device model obtained through training is used for detecting a reference map to obtain reference map information, and the reference map information is used for representing the position of at least one device in the reference map. The equipment model is used for detecting an equipment contour contained in the picture; intercepting a video stream shot by the target shooting equipment to obtain a real-time image; detecting whether equipment contour information exists in the real-time image or not based on an equipment model obtained through training; and when the equipment contour information exists in the real-time image, matching the real-time image with the reference image based on the equipment contour information in the real-time image and the reference image information to obtain a matching result. The real-time matching mechanism can quickly respond to the position change of the camera, and can immediately give an alarm or carry out correction once deviation is found, so that the accuracy and the real-time performance of detection are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an offset detection method, system, device and storage medium. Background Art

[0002] With the comprehensive construction of the power industry, in order to better monitor the production equipment in the station, video surveillance has become an indispensable part of security prevention. And its role in the operation of the station will be deepened step by step. The deployment of video surveillance points needs to be spread throughout the station. Whether the video images of the camera are accurate is related to the safety of the power grid operation. Under the influence of the external environment and human malicious damage, the camera will have problems such as video offset. If the camera is offset, the key area that should have been monitored may become invisible, forming a monitoring blind spot. In the power industry, these blind spots may include important areas such as high-voltage equipment areas, transformer rooms, and switch cabinets. Once a fault or abnormality occurs, it will not be captured and processed in time, increasing the risk of accidents.

[0003] Due to the large number of monitoring points, which usually need to cover the entire power station area, manually checking the working status of each camera one by one is not only time-consuming and labor-intensive, but also inefficient. Therefore, it is usually not possible to detect the deviation of the camera position in time. This may lead to safety hazards that are not detected for a long time. Summary of the invention

[0004] Based on the above problems, the present application provides an offset detection method, system, device and storage medium for detecting in real time whether the camera position is offset.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] The first aspect of the present application provides an offset detection method, comprising:

[0007] Using the trained device model to detect the reference image, obtaining reference image information, wherein the reference image information is used to characterize the location of at least one device in the reference image, and the device model is used to detect the device outline contained in the image;

[0008] Intercept the video stream captured by the target shooting device to obtain a real-time image;

[0009] Based on the trained device model, detecting whether the real-time image contains device profile information;

[0010] When the real-time image has device profile information, the real-time image is matched with the reference image based on the device profile information in the real-time image and the reference image information to obtain a matching result, and the matching result is used to indicate whether the target shooting device is offset.

[0011] In a possible implementation, detecting the reference image using the trained device model to obtain the reference image information includes:

[0012] The trained device model is used to detect the device in the reference image, coordinate information of the device outline in the reference image is extracted, and the coordinate information of the device outline is saved as the reference image information.

[0013] In a possible implementation, the method further includes:

[0014] When the device model does not detect the device in the reference image, the positions of the key points in the reference image are marked, and the coordinate information of the marked key points is used as the reference image information.

[0015] In a possible implementation, marking the positions of key points in the reference image and using the key point coordinate information obtained by marking as the reference image information includes:

[0016] Use a rectangular frame to mark the position of the equipment in the reference image so that the rectangular frame covers the outline of the equipment;

[0017] Record the coordinates of the upper left corner and the lower right corner of each rectangular box as the reference image information.

[0018] In a possible implementation, the method further includes:

[0019] When the real-time graph does not contain device profile information, the video stream of the target device is intercepted to obtain the real-time graph and subsequent steps are performed until the real-time graph contains device profile information.

[0020] In a possible implementation, the method further includes:

[0021] When the matching result indicates that the error between the real-time image and the reference image is greater than a preset threshold, the shooting angle of the target shooting device is adjusted, and the video stream captured by the target shooting device is intercepted to obtain a real-time image and subsequent steps until the matching result indicates that the error between the real-time image and the reference image is not greater than the preset threshold.

[0022] In a possible implementation, the device model is determined by:

[0023] Obtain several scene graphs of the target site;

[0024] Perform semantic segmentation and annotation on the devices in the acquired scene graph to obtain an annotated scene graph;

[0025] The prepared model is trained based on the annotated scene graph until a device model for detecting device contours in the image is obtained, the accuracy of which meets a preset standard.

[0026] A second aspect of the present application provides an offset detection system, comprising:

[0027] A first detection unit, configured to detect a reference image using a trained device model to obtain reference image information, wherein the reference image information is used to characterize a location of at least one device in the reference image, and the device model is used to detect a device outline contained in the image;

[0028] An interception unit, used to intercept the video stream captured by the target shooting device to obtain a real-time image;

[0029] A second detection unit is used to detect whether there is equipment profile information in the real-time image based on the trained equipment model;

[0030] A matching unit is used to match the real-time image with the reference image based on the device profile information in the real-time image and the reference image information when the real-time image has device profile information, so as to obtain a matching result, wherein the matching result is used to indicate whether the target shooting device is offset.

[0031] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the offset detection method described in the first aspect is implemented.

[0032] The fourth aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the offset detection method as described in the first aspect above.

[0033] Compared with the prior art, this application has the following beneficial effects:

[0034] The reference image is detected by the trained device model to obtain the reference image information. The reference image information can accurately represent the position of at least one device in the reference image. The position of the device can be determined by the machine learning model to significantly improve the accuracy. When the device outline is detected in the real-time image, the device outline information in the real-time image is matched with the reference image information. By comparing the position difference of the device outline in the two images, it can be concluded whether the target camera has shifted. If the matching result shows that the position of the device outline has changed significantly, it is considered that the camera has shifted. Traditional manual monitoring of camera offset relies on manual regular inspection and image comparison. This method is not only time-consuming and labor-intensive, but also easily affected by human factors, resulting in inaccurate or delayed monitoring results. However, this application realizes the automated analysis of the camera shooting picture by introducing the trained device model, which greatly improves the efficiency and accuracy of monitoring. In addition, this application not only detects the device outline information in the real-time image, but also matches the real-time image with the reference image to determine in real time whether the camera has shifted. This real-time matching mechanism can quickly respond to changes in the position of the camera. Once a shift is found, an alarm can be immediately issued or corrected, avoiding the situation where the shift problem has not been discovered and handled for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flow chart of an offset detection method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a process for determining reference image information provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a process of comparing a real-time image with a reference image provided in an embodiment of the present application;

[0039] Figure 4 A structural diagram of an offset detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0042] As mentioned above, if the camera is offset, the key areas that should be monitored may become invisible, forming monitoring blind spots. In the power industry, these blind spots may include important areas such as high-voltage equipment areas, transformer rooms, and switch cabinets. Once a failure or abnormality occurs, it will not be captured and processed in time, increasing the risk of accidents. In addition, the offset camera may capture images of non-critical areas, causing system false alarms, while real safety hazards are missed due to inadequate monitoring. This information asymmetry will interfere with the judgment of operation and maintenance personnel and delay emergency response.

[0043] The above manual inspection has the following disadvantages:

[0044] First, due to the large area of ​​the power station and the large number of cameras, manually checking the working status of each camera one by one is not only time-consuming and laborious, but also inefficient. Especially in bad weather or at night, the difficulty and cost of inspection increase significantly.

[0045] Secondly, manual inspection often relies on the experience and judgment of the inspector, which is subjective and inconsistent. Different inspectors may have different judgment criteria for the degree of deviation of the same camera, affecting the accuracy of the inspection results.

[0046] Thirdly, manual inspection is periodic and cannot monitor the camera status in real time. This means that if the camera is offset during the period between two inspections, it will not be discovered and handled in time.

[0047] In order to solve the above problems, an offset detection method is provided in an embodiment of the present application, which uses a pre-trained device model to automatically detect the reference image (i.e., the standard image captured after the camera is installed and calibrated) and the real-time image (the image in the video stream currently captured by the camera). This automated method avoids the subjectivity and fatigue factors of manual monitoring and improves the accuracy and stability of detection.

[0048] The device model is trained to accurately identify the device outline information in the image. This detection method based on image recognition technology is more detailed and accurate than manual observation and can capture tiny offset changes.

[0049] By intercepting and processing the video stream in real time, the system can instantly obtain and analyze the camera's shooting status. This real-time feature ensures that deviation problems can be discovered in time, so that corrective measures can be taken quickly to reduce problems such as blind spots or image quality degradation caused by deviation.

[0050] By matching the device profile information in the real-time image with the reference image information, the system can intelligently determine whether the camera has shifted. This matching algorithm not only takes into account the device's location information, but also multi-dimensional features such as shape, size, and angle, thereby improving the accuracy and reliability of the judgment.

[0051] Furthermore, when manually monitoring camera offset, errors may occur due to negligence, fatigue, or inconsistent judgment criteria. The solution provided in the embodiment of the present application realizes offset monitoring based on automation and intelligent technology, reduces interference from human factors, and improves the accuracy and efficiency of monitoring.

[0052] Therefore, the embodiments of the present application effectively solve the problems of inaccurate and untimely offset of manual monitoring cameras in the prior art by introducing automated and intelligent image recognition technology and real-time matching, thereby improving the stability and reliability of the monitoring process.

[0053] It should be noted that the offset detection method, system, device and storage medium provided in the present application can be applied to the field of computer technology. The above is only an example and does not limit the application field of the offset detection method, system, device and storage medium provided in the present application. In addition, the embodiments of the present application may not limit the execution subject of the offset detection. For example, the offset detection method of the embodiment of the present application can be applied to data processing devices such as terminal devices or servers. Among them, the terminal device can be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cloud server, or a cluster server composed of multiple servers.

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The following describes an offset detection method provided by the present application through an embodiment. Figure 1 ,Should Figure 1 A flow chart of an offset detection method provided in an embodiment of the present application, the method comprising:

[0056] S101. Use the trained device model to detect the reference image to obtain reference image information.

[0057] The reference image information is used to characterize the location of at least one device in the reference image, that is, to obtain accurate location information of the device in the reference image as a reference for subsequent comparison with the real-time monitoring image of the target shooting device.

[0058] See also Figure 2 , Figure 2 Schematic diagram of the reference image information determination process provided in the embodiment of the present application. Before starting the offset detection, it is necessary to obtain the reference image under the standard camera angle, collect equipment pictures in the substation, perform semantic segmentation and annotation and training, and obtain the equipment model.

[0059] In a possible implementation, the step S101 also includes: obtaining a reference image through a target shooting device. The acquisition process may include: adjusting the camera to a predetermined standard angle and focal length to ensure that the captured image is clear and completely covers the target area. Using the camera to shoot a picture whose quality meets the preset standard and save it to a computer. Performing necessary image preprocessing on the reference image, such as cropping, scaling, denoising, etc., to improve the accuracy of subsequent analysis. So that the preprocessed reference image can be analyzed and detected by the device model obtained by training.

[0060] For example, image processing software such as Photoshop, GNU Image Manipulation Program (GIMP) or programming libraries such as Open Source Computer Vision Library (OpenCV) can be used to pre-process the captured image to obtain a processed reference image, cut off unnecessary edges, and adjust the image size to meet the analysis requirements.

[0061] The device model is used to detect the device outline contained in the image. In a possible implementation, the process of determining the device model includes:

[0062] Step A1: Collect high-quality images of the substation under various environmental conditions (such as different lighting, weather, seasons, etc.). These images should include all key equipment in the substation, such as transformers, circuit breakers, disconnectors, etc.

[0063] In actual application scenarios, you can use a professional camera or a high-resolution mobile phone camera to shoot. Make sure to shoot from a variety of angles to cover all sides of the device. And capture the details and features of the device during the shooting process to facilitate subsequent semantic segmentation and annotation.

[0064] Step A2: Accurately semantically segment and annotate the key devices in the collected images. This means that each pixel in the image needs to be classified into different categories, such as the device itself, the background, other non-key objects, etc.

[0065] Use professional image annotation tools, such as the graphical image annotation software LabelMe, VGG Image Annotator (VIA), etc. to perform annotation. Accurately depict the device outline to ensure clear edges. In actual application scenarios, you can obtain manual annotation information provided by management personnel and combine the manual annotation information with the information provided by the image annotation tool to improve the accuracy of the annotation.

[0066] Step A3: Use the labeled image data to train a machine learning model that can analyze and detect the outlines of key devices in the image. Choose a suitable machine learning framework and library (such as TensorFlow, PyTorch, etc.). Design or select a neural network architecture suitable for semantic segmentation tasks (such as U-Net, DeepLabV3+, etc.). The Chinese name of TensorFlow can be called "Tensor Flow" or "Tensor Framework". PyTorch can be translated as Flying Paddle, which is the successor of Torch and is known for its excellent flexibility and support for dynamic graphs. It was developed by Facebook Artificial Intelligence Research Institute (FAIR) and is one of the most popular deep learning frameworks. U-Net can be translated as "U-shaped network" or "U-shaped convolutional neural network". DeepLabV3+ can be called "DeepLab Third Edition Enhanced Edition" or "DeepLabV3+" for short.

[0067] The labeled image data is divided into training set, validation set and test set. The model is trained using the training set data, and the network parameters and optimizer settings are continuously adjusted to improve the model performance. The validation set is used to evaluate the model performance and fine-tune it as needed. Finally, the test set is used to verify the generalization ability of the model.

[0068] After training is completed, a machine learning model that can accurately identify and output the outlines of key equipment in the substation will be obtained, namely the equipment model. This equipment model can be used in the subsequent real-time monitoring process to detect equipment status and monitor equipment position changes by automatically analyzing images captured by the camera.

[0069] In one possible implementation, after the device model is trained, the model performance can be tested periodically and the hyperparameters can be adjusted to optimize the model to ensure that the device model used in subsequent monitoring processes is always accurate.

[0070] In a possible implementation, the detecting the reference image by using the trained device model to obtain the reference image information includes:

[0071] The trained device model is used to detect the device in the reference image, coordinate information of the device outline in the reference image is extracted, and the coordinate information of the device outline is saved as the reference image information.

[0072] In a possible implementation, when the device model does not detect the device in the reference image, positions of key points in the reference image are marked, and coordinate information of the marked key points is used as reference image information.

[0073] When the device information cannot be automatically detected from the reference image in the video stream, key points in the image can be marked manually.

[0074] In the case where it is determined that there is no equipment in the reference image, the key points may be fixed feature points in the image, such as the junction of walls, doors and windows, electric poles, etc. Some stable, easy-to-identify points that will not move during the entire monitoring process can be selected as key points.

[0075] Use a specific marking method (such as a circle or a cross) to mark the key points. Record the coordinate information of each key point. Save the coordinate information of each key point into a file as the reference map information for subsequent comparison and analysis.

[0076] In a possible implementation, the selected key point may be a device in the reference map. The marking of the position of the key point in the reference map and using the coordinate information of the key point obtained by marking as the reference map information include:

[0077] Use a rectangular frame to mark the position of the device in the reference image so that the rectangular frame covers the outline of the device; record the coordinates of the upper left corner and the lower right corner of each rectangular frame as the reference image information.

[0078] Use a graphical tool or programming interface to mark the device location in the image. Choose a tool or library that supports rectangular marking. Load the previously captured real-time image into the marking tool. Manually draw a rectangular box on the real-time image to mark the location of the device. The rectangular box should cover the device outline as accurately as possible.

[0079] Record the coordinate information of each rectangular box, such as the coordinates of the upper left corner (x1, y1) and the coordinates of the lower right corner (x2, y2). You can also further record the corresponding device type or other identification information. This information can be saved as structured data, such as a comma-separated value file (CSV) or JSON format, as reference image information for subsequent matching and comparison. JSON is a text-based data representation that is easy for people to read and write, and easy for machines to parse and generate. Data in JSON format consists of key-value pairs, which are separated by commas and keys and values ​​are separated by colons.

[0080] S102: intercepting the video stream captured by the target shooting device to obtain a real-time image.

[0081] The target shooting device may be a video surveillance camera. The video stream may be video data captured by the camera in real time. The real-time image may be a single-frame image captured from the video stream for comparison with the reference image.

[0082] For more information about the process of obtaining the real-time image and comparing the real-time image with the reference image, please refer to Figure 3 , Figure 3 A schematic diagram of the process of comparing the real-time image with the reference image provided in the embodiment of the present application. The process includes obtaining the video stream and cutting frames, outputting the real-time image; using the device model to perform semantic segmentation detection on the real-time image, and judging whether the reference image contains key point information, and when the reference image contains key point information, matching the real-time image with the reference image; when the error is less than the preset threshold, turning off the video; when the error is not less than the preset threshold, turning the camera, and re-determining the real-time image for subsequent processes.

[0083] In a possible implementation, a captured video stream is obtained from a target shooting device through a network protocol, such as Real Time Streaming Protocol (RTSP) or Hypertext Transfer Protocol (HTTP).

[0084] In actual application scenarios, the video stream is checked regularly and key frames are captured as needed. The capture of key frames can be based on time intervals (such as one frame per second), video content changes (such as frame difference method), or other custom logic.

[0085] S103: Based on the trained device model, detect whether the real-time image contains device profile information.

[0086] The device model has been trained to identify and locate the device outline in the image. The device model processes the input real-time image and identifies the device outline in the image through its internal learned feature representation. For the real-time image corresponding to the captured key frame, the trained device model is used to detect the device outline.

[0087] When the real-time image contains device profile information, it indicates that the current real-time image is an image that can be used for matching and comparison, and S104 and subsequent steps are executed.

[0088] In another possible implementation, the device model can also detect the confidence of the device profile, and determine whether there is valid device profile information in the real-time map based on the output result of the device model. If the model successfully detects the device profile and its confidence meets the preset threshold requirement, it is considered that the device profile information exists in the real-time map, and S104 and subsequent steps are executed.

[0089] When the real-time graph does not contain device profile information, the video stream of the target device is intercepted to obtain the real-time graph and subsequent steps are performed until the real-time graph contains device profile information.

[0090] If there is no device outline information in the real-time image, a new key frame is captured from the video stream again as the real-time image. The trained device model is used to detect the device outline of the new real-time image. According to the detection result, it is determined whether there is device outline information in the real-time image. If not, it is returned to S102 to capture the video stream captured by the target shooting device to obtain a real-time image, and the real-time image is re-captured and detected. This cycle will continue until valid device outline information exists in a certain captured real-time image.

[0091] S104: When the real-time image has device profile information, the real-time image is matched with the reference image based on the device profile information in the real-time image and the reference image information to obtain a matching result.

[0092] The device contour information of the real-time image may include location information of the device contour (such as coordinates of a bounding box) and a possible confidence score, indicating the possibility that the detected contour is the target device.

[0093] The matching result can be the overall error calculated after comparing and matching the two images. That is, the overall error between the real-time image and the reference image is calculated based on the results of the device contour detection and the results of key point matching. This error can be based on a variety of factors, such as differences in contour position, size, shape, and the accuracy of key point matching.

[0094] In a possible implementation, feature detection algorithms in OpenCV can be used to extract key points and their descriptors from the real-time image and the reference image, such as scale-invariant feature transform SIFT, accelerated robust feature SURF, ORB, etc. Then, feature matching algorithms are used to match the two sets of key points, such as Fast Library for Approximate Nearest Neighbors (FLANN), Brute-Force Matcher (BFMatcher), etc. Finally, the transformation matrix (such as affine transformation, perspective transformation, etc.) between the real-time image and the reference image can be calculated to evaluate whether the camera has shifted. Among them, the ORB algorithm can be translated into directional fast and rotational binary robust independent features according to its full name "Oriented FAST and Rotated BRIEF" and its characteristics.

[0095] In a possible implementation, the method further includes:

[0096] When the matching result indicates that the error between the real-time image and the reference image is greater than a preset threshold, the shooting angle of the target shooting device is adjusted, and the video stream captured by the target shooting device is intercepted to obtain a real-time image and subsequent steps until the matching result indicates that the error between the real-time image and the reference image is not greater than the preset threshold.

[0097] The preset threshold can be adaptively set and adjusted according to actual needs and application scenarios, for example, it can be set to 5%. If the error is ≤5%, it is determined that the video image has no offset, the currently processed video stream is closed, and the process ends.

[0098] If the error is greater than 5%, the camera position is adjusted through the ONVIF protocol and the detection and matching process is repeated until the error returns to within 5%. When the error returns to within 5%, the camera rotation is stopped and the video stream is closed.

[0099] Exemplarily, the ONVIF protocol is used to send a control command to the camera to rotate it at a preset angle or direction. During the rotation of the camera, each frame of the camera image is continuously captured as a real-time image. The steps of device contour detection, key point matching, error calculation and judgment are repeated for each frame of the real-time image. If in a certain iteration, the error returns to the 5% range, it is determined that the video image has no offset, the currently processed video stream is closed, and the process ends. If the error still exceeds 5%, continue to rotate the camera and capture a new real-time image for iteration. Among them, the ONVIF protocol can be translated as "Open Network Video Interface Protocol" or "Open Network Video Surveillance Protocol".

[0100] In summary, the method provided in this application has the following beneficial effects:

[0101] Traditional manual monitoring of camera offset relies on regular manual inspection and image comparison, which is not only time-consuming and labor-intensive, but also easily affected by human factors, resulting in inaccurate or delayed monitoring results. This application introduces the trained device model to achieve automated and intelligent analysis of camera images, greatly improving the efficiency and accuracy of monitoring.

[0102] The device model has been trained to accurately identify and extract device outline information in the image. This high-precision detection capability ensures that no matter how the camera position changes, as long as the device is still within the field of view, it can be accurately captured and identified. This provides a reliable data basis for subsequent offset judgment.

[0103] This application not only detects the device profile information in the real-time image, but also determines in real time whether the camera is offset by matching the real-time image with the reference image. This real-time matching mechanism can quickly respond to changes in the camera's position, and once an offset is found, an alarm can be immediately issued or correction can be made, avoiding the situation where the offset problem is not discovered and handled for a long time.

[0104] The entire monitoring process requires almost no human involvement, reducing the impact of human factors on the monitoring results. At the same time, the automated monitoring process also reduces the reliance on professionals, making monitoring work more popular and efficient.

[0105] Since the device model is detected and matched based on the device outline in the image, this method is adaptable to cameras of different types and sizes. As long as the device outline can be clearly identified, offset monitoring can be performed.

[0106] The above are some specific implementations of the offset detection method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding system for offset detection. The system provided in the embodiment of the present application will be introduced from the perspective of functional modularization. Figure 4 A structural diagram of an offset detection system provided in an embodiment of the present application.

[0107] The system comprises:

[0108] A first detection unit 110 is used to detect the reference image using the trained device model to obtain reference image information, where the reference image information is used to represent the location of at least one device in the reference image, and the device model is used to detect the device outline contained in the image;

[0109] The interception unit 111 is used to intercept the video stream captured by the target shooting device to obtain a real-time image;

[0110] A second detection unit 112 is used to detect whether there is device profile information in the real-time image based on the trained device model;

[0111] The matching unit 113 is used to match the real-time image with the reference image based on the device profile information in the real-time image and the reference image information when the real-time image has device profile information, to obtain a matching result, wherein the matching result is used to indicate whether the target shooting device is offset.

[0112] In a possible implementation manner, the first detection unit is specifically configured to:

[0113] The trained device model is used to detect the device in the reference image, coordinate information of the device outline in the reference image is extracted, and the coordinate information of the device outline is saved as the reference image information.

[0114] In a possible implementation, the system further includes:

[0115] The marking unit is used to mark the positions of key points in the reference image when the device model does not detect the device in the reference image, and use the marked key point coordinate information as the reference image information.

[0116] In a possible implementation, marking the positions of key points in the reference image and using the key point coordinate information obtained by marking as the reference image information includes:

[0117] Use a rectangular frame to mark the position of the equipment in the reference image so that the rectangular frame covers the outline of the equipment;

[0118] Record the coordinates of the upper left corner and the lower right corner of each rectangular box as the reference image information.

[0119] In a possible implementation, the system further includes:

[0120] The first iteration unit is used for, when the real-time graph does not contain device profile information, executing the step of intercepting the video stream of the target device to obtain the real-time graph and subsequent steps until the real-time graph contains device profile information.

[0121] In a possible implementation, the system further includes:

[0122] The second iteration unit is used to adjust the shooting angle of the target shooting device and execute the interception of the video stream captured by the target shooting device to obtain the real-time image and subsequent steps when the matching result indicates that the error between the real-time image and the reference image is greater than a preset threshold, until the matching result indicates that the error between the real-time image and the reference image is not greater than the preset threshold.

[0123] In a possible implementation, the device model is determined by:

[0124] Obtain several scene graphs of the target site;

[0125] Perform semantic segmentation and annotation on the devices in the acquired scene graph to obtain an annotated scene graph;

[0126] The prepared model is trained based on the annotated scene graph until a device model for detecting device contours in the image is obtained, the accuracy of which meets a preset standard.

[0127] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the offset detection method provided in the embodiments of the present application.

[0128] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the offset detection method described in any embodiment of the present application.

[0129] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the offset detection method described in any embodiment of the present application.

[0130] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0131] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0132] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0133] It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0134] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting an offset, characterized in that: include: Using the trained device model to detect the reference image, obtaining reference image information, wherein the reference image information is used to characterize the location of at least one device in the reference image, and the device model is used to detect the device outline contained in the image; Intercept the video stream captured by the target shooting device to obtain a real-time image; Based on the trained device model, detecting whether the real-time image contains device profile information; When the real-time image has device profile information, the real-time image is matched with the reference image based on the device profile information in the real-time image and the reference image information to obtain a matching result, and the matching result is used to indicate whether the target shooting device is offset.

2. The method according to claim 1, characterized in that The method of detecting the reference image by using the trained device model to obtain the reference image information includes: The trained device model is used to detect the device in the reference image, coordinate information of the device outline in the reference image is extracted, and the coordinate information of the device outline is saved as the reference image information.

3. The method according to claim 2, characterized in that The method further comprises: When the device model does not detect the device in the reference image, the positions of the key points in the reference image are marked, and the coordinate information of the marked key points is used as the reference image information.

4. The method according to claim 3, characterized in that The marking of the positions of the key points in the reference image and using the coordinate information of the key points obtained by marking as the reference image information includes: Use a rectangular frame to mark the position of the equipment in the reference image so that the rectangular frame covers the outline of the equipment; Record the coordinates of the upper left corner and the lower right corner of each rectangular box as the reference image information.

5. The method according to claim 1, characterized in that The method further comprises: When the real-time graph does not contain device profile information, the video stream of the target device is intercepted to obtain the real-time graph and subsequent steps are performed until the real-time graph contains device profile information.

6. The method according to claim 1, characterized in that The method further comprises: When the matching result indicates that the error between the real-time image and the reference image is greater than a preset threshold, the shooting angle of the target shooting device is adjusted, and the video stream captured by the target shooting device is intercepted to obtain a real-time image and subsequent steps until the matching result indicates that the error between the real-time image and the reference image is not greater than the preset threshold.

7. The method according to claim 1, characterized in that The device model is determined in the following manner: Obtain several scene graphs of the target site; Perform semantic segmentation and annotation on the devices in the acquired scene graph to obtain an annotated scene graph; The prepared model is trained based on the annotated scene graph until a device model for detecting device contours in the image is obtained, the accuracy of which meets a preset standard.

8. A deviation detection system, characterized in that: The system comprises: A first detection unit, configured to detect a reference image using a trained device model to obtain reference image information, wherein the reference image information is used to characterize a location of at least one device in the reference image, and the device model is used to detect a device outline contained in the image; An interception unit, used to intercept the video stream captured by the target shooting device to obtain a real-time image; A second detection unit is used to detect whether there is equipment profile information in the real-time image based on the trained equipment model; A matching unit is used to match the real-time image with the reference image based on the device profile information in the real-time image and the reference image information when the real-time image has device profile information, so as to obtain a matching result, wherein the matching result is used to indicate whether the target shooting device is offset.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the offset detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the offset detection method according to any one of claims 1 to 7.

Images