Scene change detection method and device, computer equipment and readable storage medium
By acquiring and comparing real-time and historical data of the target scene, using loop detection and depth information differences, the problems of lighting changes in the existing technology and high computing power demand are solved, and accurate and low computing power scene change detection is achieved.
Patent Information
- Application Number
- CN202411163716.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When the prior art determines whether the dynamic target scenario changes, it is greatly affected by lighting changes and has high requirements for hardware computing power, making it difficult to achieve accurate and low computing power detection.
By obtaining real-time data and historical data of the target scene, loopback detection is used to obtain the feature point set of the current frame image and the reference frame image, the target reference point set and the current point set are determined, and the depth information difference is calculated to judge the scene changes and reduce the calculation amount.
It improves the accuracy of scene change detection, avoids the impact of lighting changes, and reduces the requirements for hardware computing power, achieving accurate detection under low computing power.
Smart Images

Figure CN119007116A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of monitoring technology, and in particular to a scene change detection method, device, computer equipment and computer-readable storage medium. Background Art
[0002] For patrols in target scenes with fewer dynamic targets, such as warehouses and field military areas, existing technologies can use patrol equipment such as unmanned vehicles and drones to obtain images of target scenes, and then perform spectral analysis based on the changes in illumination of the target scene, and use neural network detection and other methods to determine whether the target scene has changed, thereby achieving patrols. However, due to the large difference in light between day and night, methods such as spectral analysis are not effective in determining whether the target scene has changed. At the same time, methods such as neural network detection have high requirements for hardware computing power. Therefore, how to accurately determine whether the target scene has changed with lower computing power is a problem that needs to be solved. Summary of the invention
[0003] Based on this, it is necessary to provide a scene change detection method, device, computer equipment and computer-readable storage medium to address the above technical problems, so as to reduce the requirements on computing power while accurately determining whether the target scene has changed.
[0004] In a first aspect, the present application provides a scene change detection method, which is applied to patrol equipment, and the method comprises:
[0005] Obtain real-time and historical data of the target scene;
[0006] Based on loop detection, a feature point set of the current frame image in the real-time data is obtained as the current frame point set, and a reference frame point set corresponding to the current frame point set in the historical data is obtained;
[0007] When a loop is detected, the corresponding target reference point set is determined based on the matching between the current frame point set and the reference frame point set; and the corresponding target current point set is obtained based on the target reference point set;
[0008] Determine the current depth information corresponding to the current point set of the target based on the real-time data; and determine the reference depth information corresponding to the reference point set of the target based on the historical data;
[0009] Based on the difference between the current depth information and the reference depth information, the change of the target scene is determined.
[0010] In one of the embodiments, determining a corresponding target reference point set based on a matching condition between a current frame point set and a reference frame point set includes:
[0011] Based on the matching situation between the current frame point set and the reference frame point set, a target reference first point set and a target reference second point set are obtained;
[0012] Based on the target reference point set, obtain the corresponding target current point set, including:
[0013] From the current frame point set, determine an area matching the target reference first point set as the target current first point set of the target current point set;
[0014] Based on the relative position between the target reference first point set and the target reference second point set, a translation vector is obtained; based on the target current first point set and the translation vector, a target current second point set of the target current point set is obtained.
[0015] In one of the embodiments, based on the matching of the current frame point set and the reference frame point set, a target reference first point set and a target reference second point set included in the target reference point set are obtained, including:
[0016] Based on the matching between the current frame point set and the reference frame point set, a reference matching point set and a reference non-matching point set are obtained;
[0017] Based on the density of feature points and the preset size area, the reference matching point set and the reference non-matching point set are cropped to obtain a target reference first point set and a target reference second point set.
[0018] In one of the embodiments, the reference depth information includes a reference first depth value corresponding to a target reference first point set, and a reference second depth value corresponding to a target reference second point set; the current depth information includes a current first depth value corresponding to a target current first point set, and a current second depth value corresponding to a target current second point set; based on the difference between the current depth information and the reference depth information, determining the change of the target scene, including: determining the difference between the reference first depth value and the current first depth value as a first difference; determining the difference between the reference second depth value and the current second depth value as a second difference; and determining the change of the target scene based on the relationship between the first difference and the second difference.
[0019] In one of the embodiments, the real-time data also includes point cloud information of the current frame;
[0020] Determining the current depth information corresponding to the current point set of the target according to the real-time data includes: determining the field of view area corresponding to the current point set of the target as the first field of view area based on the position of the current point set of the target in the current frame image; expanding the first field of view area according to a preset ratio to obtain a second field of view area;
[0021] Point cloud information matching the second field of view area in the current frame point cloud information is extracted to obtain depth information corresponding to the target current frame point set as current depth information.
[0022] In one of the embodiments, based on the difference between the current depth information and the reference depth information, the change of the target scene is determined, including: constructing a coordinate system in the target reference point set and the target current point set based on the same standard, and selecting multiple reference points with the same coordinates from the coordinate system; according to the depth information corresponding to the multiple reference points, respectively obtaining a current average depth value and a reference average depth value; based on the difference between the current average depth value and the reference average depth value, the change of the target scene is determined.
[0023] In one embodiment, after determining the change of the target scene, the method further includes:
[0024] The current frame image is used as the historical frame image; the point cloud information in the real-time data is projected onto the historical frame image to obtain the historical frame depth image; and the historical data is updated according to the historical frame depth image.
[0025] In a second aspect, the present application further provides a scene change detection device, comprising:
[0026] The acquisition module is used to obtain real-time data and historical data of the target scene;
[0027] The analysis module is used to obtain the feature point set of the current frame image in the real-time data as the current frame point set and the reference frame point set corresponding to the current frame point set in the historical data based on loop detection; when a loop is detected, the corresponding target reference point set is determined based on the matching between the current frame point set and the reference frame point set; and the corresponding target current point set is obtained based on the target reference point set; the current depth information corresponding to the target current point set is determined according to the real-time data; and the reference depth information corresponding to the target reference point set is determined according to the historical data;
[0028] The determination module is used to determine the change of the target scene based on the difference between the current depth information and the reference depth information.
[0029] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.
[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.
[0031] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0032] The above-mentioned scene change detection method, device, computer equipment, computer-readable storage medium and computer program product have certain effects in both detection accuracy and reducing the computing power required for detection; specifically, in terms of detection accuracy: by detecting the loop, the difference between the current depth information and the reference depth information is determined, so as to detect the target scene change, which avoids the influence of factors such as illumination changes on the target scene change detection, thereby improving the accuracy of the target scene change detection; at the same time, the target reference point set and the target current point set both contain information related to the matching situation, so that the current depth information and the reference depth information determined according to the target reference point set and the target current point set also have information related to the matching situation, thereby realizing the detection of the target scene change according to the matching situation, further improving the accuracy of the target scene change detection. In terms of reducing the computing power required for detection: determine the corresponding target reference point set by matching the current frame point set with the reference frame point set; and based on the existing point set (target reference point set), obtain the corresponding target current point set, without the need to calculate the target current point set separately, which helps to reduce the amount of calculation; at the same time, determine the target scene changes based on the difference between the current depth information and the reference depth information, which only requires less computing power to achieve. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0034] Figure 1 is a schematic diagram of a flow chart of a scene change detection method in an embodiment;
[0035] Figure 2 is another schematic diagram of a flow chart of a scene change detection method in one embodiment;
[0036] Figure 3 1 is another flow chart of a scene change detection method in one embodiment;
[0037] Figure 4 is a schematic diagram of the principle of a scene change detection method in an embodiment;
[0038] Figure 5 is another principle schematic diagram of a scene change detection method in one embodiment;
[0039] Figure 6 is another principle schematic diagram of a scene change detection method in an embodiment;
[0040] Figure 7 is another flow chart of a scene change detection method in an embodiment;
[0041] Figure 8 is a structural block diagram of a scene change detection device in one embodiment;
[0042] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] In one embodiment, Figure 1 As shown, a scene change detection method is provided, which can be applied to patrol equipment, which can be but not limited to various patrol cars and various patrol robots, robot dogs, drones, etc. At the same time, the patrol equipment can detect scene changes by patrolling, or by fixing it in a certain position (i.e. not patrolling). For example, the patrol equipment can be parked in the center of the warehouse and rotate 360 degrees to scan and detect the warehouse. The method specifically includes the following steps:
[0045] Step S101: Acquire real-time data and historical data of the target scene.
[0046] Among them, the target scene is the scene that needs to be detected for changes. For example, the target scene can be a warehouse door, and whether the warehouse door has changed is detected. If it has changed, there may be a problem with the safety of warehouse property, etc. The real-time data of the target scene, that is, the real-time data captured about the target scene, exemplarily, the real-time data may include real-time image data of the target scene, real-time point cloud data (point cloud, i.e. Point Cloud, is a three-dimensional data representation form, composed of a large number of spatial coordinate points), etc. The historical data, that is, the data about the target scene generated or obtained in the past time, exemplarily, the historical data may include historical image data of the target scene, historical point cloud data, etc. Of course, the historical data may also include a set of feature points about the target scene generated from the last or last few patrols, depth information, etc., so as to facilitate the direct call of subsequent steps without the need for repetitive calculations such as feature extraction. In addition, the acquisition of real-time data and the acquisition of historical data can be performed simultaneously, or they can be performed in a certain order or at intervals, etc., which are not limited here.
[0047] Step S102: Based on loop detection, a feature point set of the current frame image in the real-time data is obtained as the current frame point set, and a reference frame point set corresponding to the current frame point set in the historical data is obtained.
[0048] Among them, loop detection, also known as closed loop detection, refers to the ability to identify scenes that have been reached, for example, loop detection is achieved based on SLAM technology (Simultaneous Localization and Mapping technology, i.e., simultaneous positioning and map construction). The reference frame point set corresponding to the current frame point set in the historical data can be a set of feature points about the target scene generated by loop detection in the past time, or it can be historical data about the target scene captured in the past time, obtained by extracting features from the historical data through loop detection, which is not limited here. At the same time, the reference frame point set can be a set of feature points of a certain frame about the target scene in the past time, or it can be a set of feature points of several frames (or even all frames) about the target scene in the past time, or it can be a reference frame point set formed after filtering and removing the feature point set of several frames (or even all frames). Exemplarily, loop detection extracts feature points from the current frame image in the real-time data, and compares and matches the extracted feature point set with the corresponding feature point set in the historical data, for example, similarity calculation, to obtain the corresponding loop detection result.
[0049] Step S103: When a loop is detected, a corresponding target reference point set is determined based on the matching between the current frame point set and the reference frame point set; and a corresponding target current point set is obtained based on the target reference point set.
[0050] Exemplarily, the loop detection result is obtained through the aforementioned steps, and it can be determined whether there is a loop in the target scene. In the case of a loop, the target scene can be detected based on the current frame point set and the reference frame point set to detect whether it changes. Specifically, the corresponding target reference point set can be obtained based on the matching situation of the current frame point set and the reference frame point set. The matching situation here can be a matching situation formed by matching the feature points in the aforementioned loop detection. For example, the loop detection in the SLAM technology has a function of generating corresponding matching situations. Exemplarily, according to the matching situation, the current frame point set includes matching the current frame point set and not matching the current frame point set. Similarly, the reference frame point set also includes matching the reference frame point set and not matching the reference frame point set. Thus, the target reference point set and the target current point set determined subsequently can also include these two situations of matching and not matching. After obtaining the target reference point set, the corresponding target current point set can be determined according to the target reference point set.
[0051] Step S104: determining the current depth information corresponding to the target current point set according to the real-time data; and determining the reference depth information corresponding to the target reference point set according to the historical data.
[0052] Among them, depth information refers to the distance of an object in the scene relative to a device (such as a camera). Depth information can improve positioning accuracy, enable the system to understand the three-dimensional structure of the scene, and then track the movement of objects in three-dimensional space, etc. According to real-time data and historical data, current depth information and reference depth information can be obtained respectively. Exemplarily, real-time data includes point cloud data (information), and the point cloud data can be projected to the area corresponding to the current point set of the target, so as to obtain the corresponding depth information as the current depth information; similarly, reference depth information can be obtained. Of course, as mentioned above, historical data can also directly include corresponding depth information, so as to directly obtain reference depth information.
[0053] Step S105: determining a change in the target scene based on a difference between the current depth information and the reference depth information.
[0054] Exemplarily, after obtaining the current depth information and the reference depth information, the change of the target scene can be determined based on the difference between the current depth information and the reference depth information. For example, the current depth information is compared with the reference depth information to obtain the difference between the two, such as the difference between their respective depth values, so as to judge whether the target scene has changed based on the difference. For example, if the difference is greater than a certain proportion or reaches a certain standard, it can be considered that the target scene has changed; otherwise, it is considered that no change has occurred.
[0055] The scene change detection method provided in this embodiment has certain effects in both detection accuracy and reducing the computing power required for detection, as follows:
[0056] In terms of detection accuracy: by detecting the loop, the difference between the current depth information and the reference depth information is determined, thereby realizing the detection of target scene changes, which avoids the influence of factors such as illumination changes on target scene change detection, thereby improving the accuracy of target scene change detection; at the same time, both the target reference point set and the target current point set contain information related to the matching situation, so that the current depth information and the reference depth information determined according to the target reference point set and the target current point set also have information related to the matching situation, thereby realizing the detection of target scene changes according to the matching situation, further improving the accuracy of target scene change detection.
[0057] In terms of reducing the computing power required for detection: determine the corresponding target reference point set by matching the current frame point set with the reference frame point set; and based on the existing point set (target reference point set), obtain the corresponding target current point set, without the need to calculate the target current point set separately, which helps to reduce the amount of calculation; at the same time, determine the target scene changes based on the difference between the current depth information and the reference depth information, which only requires less computing power to achieve.
[0058] In one embodiment, step S103 in the above embodiment may include steps S201 to S202. Figure 2 As shown, the details are as follows:
[0059] Step S201: based on the matching situation between the current frame point set and the reference frame point set, a target reference first point set and a target reference second point set included in the target reference point set are obtained.
[0060] Exemplarily, reference may be made to the explanation of the matching situation in the foregoing embodiments, including both matching and non-matching situations. Accordingly, a target reference point set including the two situations may be obtained, namely, a target reference first point set and a target reference second point set.
[0061] Step S202: From the current frame point set, determine the area that matches the target reference first point set as the target current first point set of the target current point set; based on the relative position between the target reference first point set and the target reference second point set, obtain the translation vector; based on the target current first point set and the translation vector, obtain the target current second point set of the target current point set.
[0062] Exemplarily, after obtaining the target reference first point set and the target reference second point set, the area in the current frame point set that matches the target reference first point set can be used as the target current first point set. The matching here is based on the existing matching situation in the previous step, that is, it is performed in the matched feature point area, so the corresponding calculation amount is small. The determination of the target current second point set can be determined based on the relative position relationship between the target reference first point set and the target reference second point set. Specifically, the translation vector can be obtained through the relative position between the target reference first point set and the target reference second point set. For example, the point in the upper left corner of the target reference second point set area is translated to the point in the upper left corner of the target reference first point set area, which corresponds to the vector.
[0063] In this embodiment, the target reference first point set and the target reference second point set included in the target reference point set are obtained, and the target current first point set and the target current second point set of the target current point set are determined by utilizing the matching relationship with the target reference first point set and the relative position relationship between the target reference first point set and the target reference second point set. This can reduce the amount of calculation required to determine the target current first point set and the target current second point set. At the same time, since the target reference point set includes the target reference first point set and the target reference second point set and the target current point set includes the target current first point set and the target current second point set, this makes it possible to comprehensively detect whether the target scene has changed from the above four point sets (rather than just two point sets), which helps to improve the accuracy of the corresponding target scene change detection results.
[0064] In one embodiment, step S201 in the aforementioned embodiment may include obtaining a reference matching point set and a reference non-matching point set based on the matching between the current frame point set and the reference frame point set; and cropping the reference matching point set and the reference non-matching point set based on the feature point density and the preset size area to obtain a target reference first point set and a target reference second point set.
[0065] Among them, the density of feature points, that is, the density of feature points, is unevenly distributed in a region, so the number of feature points in different parts of the region is different, that is, there are different density of feature points; the preset size area, that is, a preset area with a certain size, about the number of preset size areas, it can be that the reference matching point set and the reference non-matching point set share a preset size area, or the reference matching point set and the reference non-matching point set each have a preset size area. Exemplarily, according to the matching situation between the current frame point set and the reference frame point set, the reference matching point set (that is, the feature point set that successfully matches in the reference frame point set) and the reference non-matching point set (that is, the feature point set that fails to match in the reference frame point set) can be obtained; then, according to the different density of feature points and the preset size area, the reference matching point set and the reference non-matching point set are cropped to obtain the target reference first point set and the target reference second point set.
[0066] In this embodiment, the two dimensions of "matching" and "non-matching" are established by referring to the matching point set and the reference non-matching point set, so as to facilitate the subsequent detection of target scene changes from these two dimensions; at the same time, the reference matching point set and the reference non-matching point set are cropped by the feature point density and the preset size area to obtain the target reference first point set and the target reference second point set, which helps to reduce the amount of calculation in subsequent steps (such as calculating the corresponding depth information), and such cropping can retain the main and key information, and has little effect on the accuracy of the corresponding target scene change detection results.
[0067] In one embodiment, the reference depth information in the aforementioned embodiment includes a reference first depth value corresponding to a reference first point set of the target, and a reference second depth value corresponding to a reference second point set of the target; the current depth information in the aforementioned embodiment includes a current first depth value corresponding to a current first point set of the target, and a current second depth value corresponding to a current second point set of the target; at the same time, the step of "determining the change of the target scene based on the difference between the current depth information and the reference depth information" in the aforementioned embodiment includes: determining the difference between the reference first depth value and the current first depth value as the first difference; determining the difference between the reference second depth value and the current second depth value as the second difference; and determining the change of the target scene based on the relationship between the first difference and the second difference.
[0068] Exemplarily, as described in the above embodiments, since there are two matching situations, matching and mismatching, correspondingly, the reference depth information may include a reference first depth value corresponding to the target reference first point set, and a reference second depth value corresponding to the target reference second point set, and the current depth information may include a current first depth value corresponding to the target current first point set, and a current second depth value corresponding to the target current second point set. Further, for the two situations of matching and mismatching, the corresponding differences, namely the first difference and the second difference, can be obtained respectively, so that it is possible to determine whether the target scene has changed based on the relationship between the first difference and the second difference. For example, the first difference cannot exceed twice the second difference. If it exceeds, it is considered that the target scene has changed.
[0069] In this embodiment, whether the target scene changes is detected by obtaining two differences (a first difference and a second difference) rather than a single difference, which helps to improve the accuracy of the corresponding target scene change detection results; at the same time, the calculation of the difference is relatively simple and does not involve complex calculation methods, and accordingly, the required computing power is also lower.
[0070] In one embodiment, the real-time data in the aforementioned embodiment also includes current frame point cloud information; the step of "determining the current depth information corresponding to the target current point set based on the real-time data" in the aforementioned embodiment includes: based on the position of the target current point set in the current frame image, determining the field of view area corresponding to the target current point set as the first field of view area; expanding the first field of view area according to a preset ratio to obtain a second field of view area; extracting the point cloud information in the current frame point cloud information that matches the second field of view area, and obtaining the depth information corresponding to the target current frame point set as the current depth information.
[0071] Among them, the current frame point cloud information refers to the point cloud data corresponding to the current frame. For example, in practice, the calibration of devices that capture target scene-related data (such as cameras and lidars, etc.) is often not ideal, so the field of view area in the image and the point cloud field of view area will not completely overlap, so it is divided into the camera field of view area and the lidar field of view area. Due to some calibration errors, if the projection is completely based on the camera field of view area, some feature points in the local field of view area corresponding to the image feature point set area may not be projected. Point clouds, there is no depth information, and in order to accurately calculate the depth information, it is also necessary to take some points in the point cloud around the feature points. These points in the local field of view edge point cloud are likely to be outside the local field of view area corresponding to the feature point set area. Based on the above reasons, the field of view area corresponding to the current point set of the target can be determined as the first field of view area according to the position of the current point set of the target in the current frame image; then the first field of view area is enlarged, and the enlargement ratio (i.e., the preset ratio) is not limited, so as to obtain the second field of view area; finally, the point cloud information matching the second field of view area can be extracted from the current frame point cloud information according to the second field of view area, so as to obtain the depth information corresponding to the current frame point set of the target as the current depth information, for example, the point cloud information matching the second field of view area is projected into the corresponding image, so that the image of the area has depth information. The reference depth information can also be obtained by the same method. Of course, the corresponding depth information can also be directly included in the historical data, so as to facilitate the direct determination of the reference depth information.
[0072] In this embodiment, considering that the equipment for collecting real-time data in reality is often not calibrated well and has calibration errors, if the corresponding point cloud information is obtained completely according to the original field of view area (first field of view area), some feature points may lack corresponding point cloud information, and thus have no depth information. In this regard, the method of this embodiment determines the field of view area (first field of view area) corresponding to the current point set of the target and expands it to obtain the second field of view area, and determines the current depth information based on the second field of view area, thereby avoiding the above-mentioned problem. At the same time, considering that some points of the point cloud around the feature points need to be taken to calculate the depth information, these points are likely to be outside the local field of view area (first field of view area) corresponding to the feature point set area. Therefore, the method of this embodiment also helps to obtain more accurate current depth information.
[0073] In one embodiment, the step of "determining changes in the target scene based on the difference between the current depth information and the reference depth information" in the aforementioned embodiment includes: constructing a coordinate system in the target reference point set and the target current point set based on the same standard, and selecting multiple reference points with the same coordinates from the coordinate system; obtaining a current average depth value and a reference average depth value according to the depth information corresponding to the multiple reference points; and determining changes in the target scene based on the difference between the current average depth value and the reference average depth value.
[0074] For example, for the area of the target reference point set and the area of the target current point set, the upper left corner is taken as the origin of the coordinate system, the horizontal left is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis coordinate system, so as to establish a coordinate system, so that the same value point in the two coordinate systems can be taken. If the first coordinate system is established in the area of the target reference point set and the second coordinate system is established in the area of the target current point set, then the coordinate point in the first coordinate system is taken as P1 (x1, y1), then the coordinate point P2 (x2, y2) in the second coordinate system has x1=x2, y1=y2, and the corresponding points of P1 and P2 in the absolute coordinate system have corresponding depth information. That is, the points with the same coordinates are taken as reference points, and the reference points have corresponding depth information. The number of reference points can be multiple, for example, four, and then the depth values corresponding to these reference points are summed and averaged, that is, the current average depth value and the reference average depth value are obtained. Finally, it is possible to determine whether the target scene has changed based on the difference between the current average depth value and the reference average depth value.
[0075] In this embodiment, a coordinate system is constructed and reference points with the same coordinates are selected, and the current average depth value and the reference average depth value are calculated according to the depth information of the reference points, which reduces the corresponding amount of calculation; and whether the target scene has changed is determined based on the difference between the current average depth value and the reference average depth value. The calculation of the difference is relatively simple, with only subtraction and no multiplication or division involved, so the corresponding amount of calculation is reduced. At the same time, the reduction in the amount of calculation has little effect on the accuracy of the corresponding target scene change detection result.
[0076] In one embodiment, after the aforementioned step of "determining the changes in the target scene", the scene change detection method also includes: projecting the point cloud information in the real-time data to the corresponding current frame image, and using it as a historical frame depth image; updating the historical data according to the historical frame depth image.
[0077] Exemplarily, after completing the detection of the target scene, the acquired real-time data can be used to update the corresponding historical data. Specifically, the point cloud information in the real-time data can be projected into the corresponding current frame image, so that the current frame image has depth information, and the projected current frame image is used as the historical frame depth image, and then the historical data can be updated according to the historical frame depth image; of course, the current frame image can also be updated to the historical frame image first, and then the relevant point cloud information is projected onto the historical frame image, so as to realize the projection of the point cloud information in the real-time data into the corresponding current frame image (historical frame image), obtain the historical frame depth image, and then update the historical data. It should be noted that the update here can include replacing the corresponding historical data with real-time data, or adding the real-time data to the historical data.
[0078] In this embodiment, the corresponding historical data is updated by using the acquired real-time data, so that the historical data has better reference accuracy, and the timeliness and accuracy of the scene change detection results are not affected by the outdated historical data.
[0079] In one embodiment, a scene change detection method is provided, which can be improved based on the existing SLAM system (Simultaneous Localization and Mapping, i.e., simultaneous positioning and mapping), for example, the ORB-SLAM system (a general and accurate monocular SLAM system); at the same time, the method can be applied to patrol equipment in places with fewer dynamic targets, such as patrol cars in warehouses and field military areas. Regarding the hardware setting of the patrol equipment, a (calibrated) camera and a laser radar are required, and the FOV (field of view) of the laser radar should be able to cover the FOV (field of view) of the camera, as well as a fill light that can cover the camera's field of view as much as possible. When the patrol equipment is powered on, the laser radar obtains point cloud data, and the camera obtains image data, which can be dedistorted and synchronized with the timestamp. The principle of the method can be summarized as follows: when there is no historical reference frame (i.e., there is no historical data about the target scene), the device can be deployed first, data (point cloud and image) can be collected, a visual SLAM map can be constructed, and reference data (i.e., historical data) can be initialized. When the device arrives at a certain place, the reference data around the place is loaded in advance. When the SLAM system detects a loop, it compares the depth difference between the feature points of the historical reference frame image and the feature points of the current frame image. If the difference is large, it is determined that the scene has changed and an alarm notification is reported. (Visual SLAM loop detection must extract feature points, so the feature points are extracted before comparing the depth difference of the feature points.) The principle of this method can be seen in Figure 7 The method includes steps S301 to S303, as shown in FIG. Figure 3As shown, the specific process steps are as follows:
[0080] Step S301: Determine the target reference point set and the target current point set. This step corresponds to "obtaining real-time data and historical data of the target scene; based on loop detection, obtaining the feature point set of the current frame image in the real-time data as the current frame point set, and the reference frame point set corresponding to the current frame point set in the historical data; when a loop is detected, determining the corresponding target reference point set based on the matching of the current frame point set and the reference frame point set; and obtaining the corresponding target current point set based on the target reference point set" and other related contents.
[0081] In step S301, steps S3011 to S3013 may be included, which are specifically as follows:
[0082] Step S3011: Get the matching situation. Match the point pairs of two frames, set the historical reference frame feature point set as Ph (i.e., reference frame point set), set the current frame feature point set as Pc (i.e., current frame point set), set the part of the historical frame point set that has been successfully matched as Phm (i.e., reference matching point set), set the part of the current frame point set that has been successfully matched as Pcm, set the unmatched historical frame feature point set as Phn (reference non-matching point set), set the unmatched current frame feature point set as Pcn, and the relationship diagram can be seen in Figure 4 ; It should be pointed out that the matching in this step is feature point matching, which belongs to the loop detection function in the SLAM system framework.
[0083] Step S3012: Calculate the most densely populated areas of Phm and Phn (square areas with the same area), recorded as Zhm (i.e., the first target reference point set) and Zhn (i.e., the second target reference point set). This step corresponds to "Based on the density of feature points and the preset size area, the reference matching point set and the reference non-matching point set are cropped to obtain the first target reference point set and the second target reference point set" and other related contents.
[0084] Step S3013: See Figure 5 As shown, calculate the position of Zhn relative to Zhm, record the translation vector Vh, then Zhn=Zhm-Vh, record the area of the current frame points that match the points in Zhm as Zcm (i.e., the current first point set of the target), and record the relative area corresponding to Zhn as Zcn=Zcm-Vh (i.e., the current second point set of the target). This step corresponds to "from the current frame point set, determine the area that matches the target reference first point set as the target current first point set of the target current point set; based on the relative position between the target reference first point set and the target reference second point set, obtain the translation vector; based on the target current first point set and the translation vector, obtain the target current second point set of the target current point set" and other related content.
[0085] Step S302: Determine the current depth information corresponding to the target current point set and the reference depth information corresponding to the target reference point set. This step corresponds to "determine the current depth information corresponding to the target current point set based on real-time data; and determine the reference depth information corresponding to the target reference point set based on historical data" and other related contents.
[0086] Regarding the determination of the current depth information: by calculating the position of Zcm and Zcn in the image field of view, the corresponding camera field of view area can be obtained. According to the camera field of view area, the corresponding lidar field of view area can be obtained by expanding the area. For example, the field of view area corresponding to the point cloud can be expanded at a certain ratio, or the camera field of view area can be expanded first, and then the corresponding lidar field of view area can be determined according to the expanded camera field of view area. The three-dimensional points of the local point cloud area (corresponding lidar field of view area) are projected into the current frame image so that the corresponding image area in the current frame image has depth information, thereby obtaining the current depth information corresponding to the current point set of the target. It should be pointed out that if the camera and lidar are calibrated ideally, the field of view area in the image and the point cloud field of view area are overlapped, but in reality the calibration is often not ideal, so it is divided into the camera field of view area and the lidar field of view area. The local area of Zcm and Zcn in the camera field of view and the corresponding lidar field of view area are the local point cloud area. The reason for the expansion processing here is that there may be calibration errors in reality. If the projection is completely based on the original field of view area, some feature points may not be projected in the local field of view area corresponding to the image feature point set area. There is no depth information, and the calculation of depth information is also to take some points of the point cloud around the feature points. These points of the local field of view edge point cloud are likely to be outside the local field of view area corresponding to the feature point set area. Therefore, expansion processing is required. Regarding the determination of reference depth information: it can be directly determined based on the data (historical data) obtained from the previous round of patrols. When not patrolling, strong real-time performance is not required, and the depth information of each feature point in all historical reference frames can be calculated.
[0087] Step S303: Determine the change of the target scene based on the difference between the current depth information and the reference depth information. This step corresponds to "determine the change of the target scene based on the difference between the current depth information and the reference depth information; construct a coordinate system in the target reference point set and the target current point set based on the same standard, and select multiple reference points with the same coordinates from the coordinate system; obtain the current average depth value and the reference average depth value based on the depth information corresponding to the multiple reference points; determine the change of the target scene based on the difference between the current average depth value and the reference average depth value" and other related contents.
[0088] Calculate the average depth of Zhm, Zhn, Zcm, and Zcn, recorded as Lhm, Lhn, Lcm, and Lcn. The calculation method is to use the same point of Zhm and Zcm as the reference coordinate system to take the depth value of the point at the same position, where the Zhm and Zcm areas are determined by four points. For example, the upper left corner is taken as the 0 point of the coordinate system, the horizontal left is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis coordinate system. This coordinate system is established to take the same value points in the two coordinate systems. If the Zhm and Zcm areas establish relative coordinate systems CShm and CScm respectively, then the coordinate point P1 (x1, y1) under CShm, then the coordinate point P2 (x2, y2) under CScm has x1=x2, y1=y2. The corresponding points of P1 and P2 in the absolute coordinate system must have corresponding depth information. You can refer to this Figure 6 As shown in the figure, the diamond points in the image area, Zhn and Zcn are the same. Regarding the calculation of the average depth, take the depth values of the same point and add them up and divide them by the number of depth values. Just add them up and divide them by the number in the ordinary way to get the average depth. Calculate the depth error Em=|Lhm–Lcm| of the matched dense feature point area, calculate the depth difference En=|Lhn–Lcn| of the dense area of the unmatched feature point set, judge the relationship between En and Em, and determine whether the scene has changed by setting rules. For example, En cannot exceed 2 times of Em. If it exceeds, it is considered that the scene has changed. In addition, when the patrol is over, all current frames replace the historical frames, and the point cloud is projected onto the image.
[0089] Through the above methods, the goal of detecting scene changes with low computing power is achieved. The entire detection calculation process is mainly based on the addition and subtraction of point sets, and multiplication and division are rarely used, which effectively reduces the amount of calculation; we only perform three-dimensional point projection on two local feature point areas and calculate the average depth, so the amount of calculation is further reduced. In terms of detection accuracy: by detecting the loop, the difference between the current depth information and the reference depth information is determined, so as to detect the change of the target scene, which avoids the influence of factors such as illumination changes on the detection of target scene changes, thereby improving the accuracy of target scene change detection; at the same time, the target scene change is detected according to the matching situation, and the target scene is comprehensively detected from four point sets (rather than only two point sets) to detect whether the target scene has changed, which improves the accuracy of the corresponding target scene change detection results. Considering that the calibration error of the device in reality and the need to take some points of the point cloud around the feature point to calculate the depth information, it is helpful to obtain more accurate current depth information by expanding the field of view area to determine the current depth information. In terms of reducing the computing power required for detection: the corresponding target reference point set is determined through matching; and based on the existing point set (target reference point set), the corresponding target current point set is obtained, and there is no need to calculate the target current point set separately, which helps to reduce the amount of calculation; at the same time, the target scene change is determined based on the difference between the current depth information and the reference depth information, which only requires less computing power to achieve. By utilizing the matching relationship with the target reference first point set and the relative position relationship between the target reference first point set and the target reference second point set, the target current first point set and the target current second point set are determined, which can reduce the amount of calculation; at the same time, the reference matching point set and the reference non-matching point set are cropped by the density of feature points and the preset size area, so as to obtain the target reference first point set and the target reference second point set, which helps to reduce the amount of calculation of subsequent steps (such as calculating the corresponding depth information), and such cropping can retain the main and key information, and has little effect on the accuracy of the corresponding target scene change detection results. By constructing a coordinate system and selecting reference points with the same coordinates, the current average depth value and the reference average depth value are calculated according to the depth information of the reference points, which reduces the corresponding amount of calculation; and, whether the target scene has changed is determined based on the difference between the current average depth value and the reference average depth value. The calculation of the difference is relatively simple, with only subtraction and no multiplication or division involved, so the corresponding amount of calculation is reduced. At the same time, the reduction in the amount of calculation has little effect on the accuracy of the corresponding target scene change detection results.
[0090] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0091] Based on the same inventive concept, the embodiment of the present application also provides a scene change detection device for implementing the scene change detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more scene change detection device embodiments provided below can refer to the limitations of the scene change detection method above, and will not be repeated here.
[0092] In an exemplary embodiment, Figure 8 As shown, a scene change detection device 800 is provided, comprising:
[0093] The acquisition module 801 is used to acquire the real-time data and historical data of the target scene;
[0094] The analysis module 802 is used to obtain, based on loop detection, a feature point set of the current frame image in the real-time data as the current frame point set, and a reference frame point set corresponding to the current frame point set in the historical data; when a loop is detected, determine the corresponding target reference point set based on the matching between the current frame point set and the reference frame point set; and obtain the corresponding target current point set based on the target reference point set; determine the current depth information corresponding to the target current point set according to the real-time data; and determine the reference depth information corresponding to the target reference point set according to the historical data;
[0095] The determination module 803 is used to determine the change of the target scene based on the difference between the current depth information and the reference depth information.
[0096] In one of the embodiments, the analysis module 802 is also used to: obtain a target reference first point set and a target reference second point set included in the target reference point set based on the matching between the current frame point set and the reference frame point set; determine from the current frame point set an area that matches the target reference first point set as the target current first point set of the target current point set; obtain a translation vector based on the relative position between the target reference first point set and the target reference second point set; and obtain a target current second point set of the target current point set based on the target current first point set and the translation vector.
[0097] In one of the embodiments, the analysis module 802 is also used to: obtain a reference matching point set and a reference non-matching point set based on the matching between the current frame point set and the reference frame point set; and crop the reference matching point set and the reference non-matching point set based on the feature point density and the preset size area to obtain a target reference first point set and a target reference second point set.
[0098] In one of the embodiments, the determination module 803 is also used for: the reference depth information includes a reference first depth value corresponding to the target reference first point set, and a reference second depth value corresponding to the target reference second point set; the current depth information includes a current first depth value corresponding to the target current first point set, and a current second depth value corresponding to the target current second point set; determining the difference between the reference first depth value and the current first depth value as the first difference; determining the difference between the reference second depth value and the current second depth value as the second difference; and determining the change of the target scene based on the relationship between the first difference and the second difference.
[0099] In one of the embodiments, the analysis module 802 is also used for: the real-time data also includes the current frame point cloud information; based on the position of the target current point set in the current frame image, determining the field of view area corresponding to the target current point set as the first field of view area; expanding the first field of view area according to a preset ratio to obtain the second field of view area; extracting the point cloud information in the current frame point cloud information that matches the second field of view area, and obtaining the depth information corresponding to the target current frame point set as the current depth information.
[0100] In one of the embodiments, the determination module 803 is also used to: construct a coordinate system in the target reference point set and the target current point set based on the same standard, and select multiple reference points with the same coordinates from the coordinate system; obtain the current average depth value and the reference average depth value according to the depth information corresponding to the multiple reference points; and determine the change of the target scene based on the difference between the current average depth value and the reference average depth value.
[0101] Each module in the scene change detection device 800 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0102] In one of the embodiments, the determination module 803 is also used to project the point cloud information in the real-time data to the corresponding current frame image after determining the change of the target scene, and use it as a historical frame depth image; and update the historical data according to the historical frame depth image.
[0103] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data required for scene change detection. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a scene change detection method is implemented.
[0104] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0105] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of any one of the methods in the foregoing embodiments when executing the computer program.
[0106] In one embodiment, when the processor executes the computer program, it also implements the steps of the method in any one of the aforementioned embodiments.
[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the aforementioned embodiments are implemented.
[0108] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the method of any one of the aforementioned embodiments when executed by a processor.
[0109] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0111] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A scene change detection method, characterized in that: Applied to patrol equipment, the method comprises: Obtain real-time and historical data of the target scene; Based on loop detection, a feature point set of a current frame image in the real-time data is obtained as a current frame point set, and a reference frame point set corresponding to the current frame point set in the historical data; In the case where a loop is detected, a corresponding target reference point set is determined based on the matching between the current frame point set and the reference frame point set; and a corresponding target current point set is acquired based on the target reference point set; Determine the current depth information corresponding to the target current point set according to the real-time data; and determine the reference depth information corresponding to the target reference point set according to the historical data; Based on the difference between the current depth information and the reference depth information, a change of the target scene is determined.
2. The method according to claim 1, characterized in that The determining a corresponding target reference point set based on a matching condition between the current frame point set and the reference frame point set includes: Based on the matching situation between the current frame point set and the reference frame point set, obtaining a target reference first point set and a target reference second point set included in the target reference point set; The acquiring a corresponding target current point set based on the target reference point set comprises: From the current frame point set, determining an area matching the target reference first point set as a target current first point set of the target current point set; Obtaining a translation vector based on a relative position between the target reference first point set and the target reference second point set; A target current second point set of the target current point set is obtained based on the target current first point set and the translation vector.
3. The method according to claim 2, characterized in that The obtaining, based on the matching condition between the current frame point set and the reference frame point set, a target reference first point set and a target reference second point set included in the target reference point set comprises: Based on the matching situation between the current frame point set and the reference frame point set, a reference matching point set and a reference non-matching point set are obtained; Based on the density of feature points and a preset size area, the reference matching point set and the reference non-matching point set are cropped to obtain a target reference first point set and a target reference second point set.
4. The method according to claim 2 or 3, characterized in that: The reference depth information includes a reference first depth value corresponding to the target reference first point set, and a reference second depth value corresponding to the target reference second point set; The current depth information includes a current first depth value corresponding to a current first point set of the target, and a current second depth value corresponding to a current second point set of the target; The determining, based on the difference between the current depth information and the reference depth information, a change of the target scene includes: Determine a difference between the reference first depth value and the current first depth value as a first difference; determine a difference between the reference second depth value and the current second depth value as a second difference; Based on the relationship between the first difference and the second difference, a change in the target scene is determined.
5. The method according to claim 1, characterized in that The real-time data also includes current frame point cloud information; The determining, according to the real-time data, current depth information corresponding to the current point set of the target includes: Based on the position of the target current point set in the current frame image, determining a field of view area corresponding to the target current point set as a first field of view area; Enlarging the first field of view area according to a preset ratio to obtain a second field of view area; Point cloud information matching the second field of view area in the current frame point cloud information is extracted to obtain depth information corresponding to the target current frame point set as current depth information.
6. The method according to claim 1, characterized in that The determining, based on the difference between the current depth information and the reference depth information, a change of the target scene includes: Based on the same standard, a coordinate system is constructed in the target reference point set and the target current point set respectively, and a plurality of reference points with the same coordinates are selected from the coordinate system; According to the depth information corresponding to the plurality of reference points, a current average depth value and a reference average depth value are obtained respectively; Based on the difference between the current average depth value and the reference average depth value, a change in the target scene is determined.
7. The method according to claim 1, characterized in that After determining the change of the target scene, the method further includes: Projecting the point cloud information in the real-time data to the corresponding current frame image and using it as a historical frame depth image; The historical data is updated according to the historical frame depth image.
8. A scene change detection device, characterized in that: The device comprises: The acquisition module is used to obtain real-time data and historical data of the target scene; An analysis module, for obtaining, based on loop detection, a feature point set of a current frame image in the real-time data as a current frame point set, and a reference frame point set corresponding to the current frame point set in the historical data; in the case where a loop is detected, determining a corresponding target reference point set based on a match between the current frame point set and the reference frame point set; and obtaining a corresponding target current point set based on the target reference point set; determining current depth information corresponding to the target current point set according to the real-time data; and determining reference depth information corresponding to the target reference point set according to the historical data; A determination module is used to determine a change in the target scene based on a difference between the current depth information and the reference depth information.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
