Residue detection methods, apparatuses, devices, and storage media

Abstract

The application provides a residual detection method, device and equipment and a storage medium. The method comprises the following steps: obtaining a target image to be detected; obtaining a relevant feature image according to a first feature map of the target image and a second feature map of a background image corresponding to the target image; and performing residual detection on the relevant feature image to obtain a residual detection result of the target image. The technical scheme can realize stable and accurate residual detection on any scene and positioning of the residual, improve the anti-interference capability of the residual detection model, reduce the interference of environmental factors such as light on the residual detection model, improve the accuracy of residual detection, and reduce the false detection probability.

Description

Technical Field

[0001] This application relates to the field of image recognition, specifically to a method, apparatus, device, and storage medium for detecting abandoned objects. Background Technology

[0002] Currently, when working in workplaces such as factory workshops, laboratories, and express delivery transit centers, employees may leave personal or work items behind. Due to the large number of items in these workplaces, lost items are difficult to recover, and they also pose certain safety hazards in these work scenarios. Existing methods for detecting lost items have poor resistance to interference, a high probability of false detection, and are unable to effectively detect lost items. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for detecting abandoned items, aiming to solve the technical problem of easy false detection in the prior art.

[0004] On the one hand, embodiments of this application provide a method for detecting abandoned items, the method comprising the following steps:

[0005] Acquire the target image to be detected;

[0006] Based on the first feature map of the target image and the second feature map of the corresponding background image, a relevant feature image is obtained;

[0007] The relevant feature images are subjected to residue detection to obtain the residue detection results of the target image.

[0008] In one possible implementation of this application, the remnant detection method is applied to a remnant detection model;

[0009] The step of obtaining a relevant feature image based on a first feature map of the target image and a second feature map of the corresponding background image includes:

[0010] The first feature map of the target image and the second feature map of the background image are input into the object detection model. The feature correlation between the first feature map and the second feature map is compared to generate a related feature image.

[0011] In one possible implementation of this application, obtaining the relevant feature image based on the first feature map of the target image and the second feature map of the corresponding background image includes:

[0012] Extract the first feature block from the first feature map and the second feature block from the second feature map corresponding to the first feature block;

[0013] Perform convolution operations on each of the first feature blocks and the corresponding second feature blocks to obtain each relevant feature;

[0014] The aforementioned correlation features are integrated to obtain a correlation feature image.

[0015] In one possible implementation of this application, the remnant detection model further includes a first feature extraction module and a second feature extraction module;

[0016] After acquiring the target image to be detected, the process further includes:

[0017] The target image to be detected is input into the first feature extraction module for feature extraction to obtain the first feature map;

[0018] The background image corresponding to the target image is input into the second feature extraction module for feature extraction to obtain the second feature map.

[0019] In one possible implementation of this application, the step of performing residue detection on the relevant feature image to obtain the residue detection result of the target image includes:

[0020] The relevant feature images are input into the object detection model, and the relevant feature images are subjected to image upscaling to obtain high-dimensional relevant feature images;

[0021] Perform convolution operation on the high-dimensional related feature image to obtain the high-dimensional convolution operation result;

[0022] The high-dimensional convolution operation result is mapped using a linear convolution machine to obtain the low-dimensional convolution operation result;

[0023] Based on the results of the low-dimensional convolution operation, the object classification and detection are performed to obtain the object detection results of the target image.

[0024] In one possible implementation of this application, the step of performing residue classification and detection based on the low-dimensional convolution operation result to obtain the residue detection result of the target image includes:

[0025] The low-dimensional convolution operation result is subjected to a classification convolution operation to obtain the change image features of the relevant feature image;

[0026] The changed image features are subjected to residue detection to obtain residue detection results, wherein the residue detection results include residues and non-residues.

[0027] In one possible implementation of this application, before performing residue detection on the relevant feature image to obtain the residue detection result of the target image, the method further includes:

[0028] The low-dimensional convolution result is subjected to a localization convolution operation to obtain the change image features of the relevant feature image;

[0029] Obtain the object detection box of the changed image features, and obtain the position information of the object detection box;

[0030] Based on the location information, the features of the changed image are used to locate the remaining objects.

[0031] In one possible implementation of this application, obtaining the object detection box of the changed image features and obtaining the position information of the object detection box includes:

[0032] Obtain the coordinates of each vertex of the object detection frame, and calculate the length and width of the object detection frame based on the vertex coordinates;

[0033] The center point coordinates of the object detection frame are calculated based on the vertex coordinates, and the center point coordinates, length, and width of the object detection frame are determined as the position information of the object detection frame.

[0034] On the other hand, this application provides a device for detecting abandoned items, the device comprising:

[0035] The image acquisition module is configured to acquire the target image to be detected;

[0036] The correlation detection module is configured to obtain a correlation feature image based on a first feature map of the target image and a second feature map of the background image corresponding to the target image;

[0037] The residue detection module is configured to perform residue detection on the relevant feature image to obtain the residue detection result of the target image.

[0038] On the other hand, this application also provides a device for detecting abandoned items, the device comprising:

[0039] One or more processors;

[0040] Memory; and

[0041] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the remnant detection method.

[0042] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the said remnant detection method.

[0043] This application acquires a target image to be detected, and separately acquires a first feature map of the target image and a second feature map of the corresponding background image. Correlation detection is then performed on the first and second feature maps to obtain related feature images. After acquiring the related feature images, object detection is performed on them to obtain the object detection result for the target image. This achieves stable and accurate object detection and localization in any scene, improves the anti-interference capability of the object detection model, reduces the interference of environmental factors such as lighting on the object detection model, improves the accuracy of object detection, and reduces the probability of false detection. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a scenario for the residue detection method according to an embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating one embodiment of the residue detection method in this application.

[0047] Figure 3 This is a flowchart illustrating an embodiment of the method for obtaining relevant feature images in the object detection method provided in this application.

[0048] Figure 4 This is a flowchart illustrating an embodiment of the residue detection method provided in this application for detecting residues in the relevant feature image to obtain the residue detection result of the target image;

[0049] Figure 5 This is a flowchart illustrating an embodiment of the method for locating debris in the debris detection method provided in this application.

[0050] Figure 6 This is a schematic diagram of an embodiment of the residue detection device provided in this application;

[0051] Figure 7 This is a schematic diagram of one embodiment of the residue detection device provided in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0054] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0055] Currently, when working in workplaces such as factory workshops, laboratories, and express delivery transit centers, employees may leave personal or work items behind. Due to the large number of items in these workplaces, lost items are difficult to recover, and they also pose certain safety hazards in these work scenarios. Existing methods for detecting lost items have poor resistance to interference, a high probability of false detection, and are unable to effectively detect lost items.

[0056] Based on this, this application proposes a method, apparatus, device, and computer-readable storage medium for detecting abandoned objects, in order to solve the technical problem that abandoned object detection is prone to false detection in the prior art.

[0057] The abandoned object detection method in this embodiment of the invention is applied to an abandoned object detection device, which is set in an abandoned object detection equipment. The abandoned object detection equipment is provided with one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the abandoned object detection method. The abandoned object detection equipment can be a smart terminal, such as a mobile phone, tablet computer, smart TV, network device, and smart computer. Optionally, the abandoned object detection equipment can also be a server or a service cluster composed of multiple servers.

[0058] like Figure 1 As shown, Figure 1 This is a schematic diagram of a scenario for a method for detecting debris according to an embodiment of this application. The scenario for detecting debris in this embodiment includes a debris detection device 100 (the debris detection device 100 integrates a debris detection apparatus), and a computer-readable storage medium corresponding to the debris detection method is running in the debris detection device 100 to execute the steps of the debris detection method.

[0059] Understandable Figure 1 The waste detection equipment in the scenario of the waste detection method shown, or the devices included in the waste detection equipment, do not constitute a limitation on the embodiments of the present invention. That is, the number or type of waste detection equipment in the scenario of the waste detection method, or the number or type of devices included in each equipment, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of the present invention.

[0060] In this embodiment of the invention, the residue detection device 100 is mainly used for: acquiring an image of the target to be detected;

[0061] Based on the first feature map of the target image and the second feature map of the corresponding background image, a relevant feature image is obtained;

[0062] The relevant feature images are subjected to residue detection to obtain the residue detection results of the target image.

[0063] The abandoned object detection device 100 in this embodiment of the invention can be an independent abandoned object detection device, such as a mobile phone, tablet computer, smart TV, network device, server and smart computer, or an abandoned object detection network or abandoned object detection cluster composed of multiple abandoned object detection devices.

[0064] This application provides a method, apparatus, device, and computer-readable storage medium for detecting abandoned items, which will be described in detail below.

[0065] It will be understood by those skilled in the art that Figure 1 The application environment shown is only one application scenario related to the solution of this application and does not constitute a limitation on the application scenario of this application. Other application environments may include more than one application scenario. Figure 1 The number of more or fewer artifact detection devices shown, or the network connections for artifact detection, for example Figure 1 Only one object detection device is shown in the image. It is understood that the scenario of this object detection method may also include one or more object detection devices, which are not specifically limited here. The object detection device 100 may also include a memory for storing surveillance video sequences and other data.

[0066] It should be noted that, Figure 1 The schematic diagram of the residue detection method shown is merely an example. The scenarios of the residue detection method described in the embodiments of the present invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention.

[0067] Based on the scenarios described above for the methods of detecting abandoned items, various embodiments of the abandoned item detection method disclosed in this invention are proposed.

[0068] like Figure 2 As shown, Figure 2 This is a flowchart illustrating one embodiment of the residue detection method in this application. The image processing method includes the following steps 201-203:

[0069] 201. Obtain the target image to be detected;

[0070] The abandoned object detection method in this embodiment is applied to an abandoned object detection device. The type and number of abandoned object detection devices are not specifically limited. That is, the abandoned object detection device can be one or more smart terminals or servers. In a specific embodiment, the abandoned object detection device is a smart computer or server that is communicatively connected to a surveillance camera.

[0071] When the lost and found detection equipment is in operation, it can receive lost and found detection requests. These requests are operational instructions that drive the equipment to detect whether any lost or found items exist in the monitored area of ​​the surveillance camera. The method of triggering these requests is not specifically limited; they can be user-initiated. For example, if a worker loses or leaves a lost or found item within the surveillance camera's range, the user can interact with the equipment. Optionally, this interaction can be via a controller or by clicking a virtual button on the equipment's display interface. Alternatively, lost and found detection requests can be automatically triggered periodically by the lost and found monitoring equipment or a connected surveillance camera according to a preset timed detection schedule.

[0072] Upon receiving a debris detection request, the debris detection device acquires an image of the target area to be detected and inputs it into a pre-set debris detection model. The model then performs debris detection on the target image to determine if any debris is present. The target image is a video frame captured by a surveillance camera connected to the debris detection device, showing the monitored area. Optionally, the debris detection model includes a first feature extraction module, a second feature extraction module, a correlation acquisition module, and a probe module. This debris detection model can be installed within the debris detection device or within a surveillance camera communicatively connected to it.

[0073] 202. Based on the first feature map of the target image and the second feature map of the background image corresponding to the target image, a relevant feature image is obtained;

[0074] After acquiring the target image to be reviewed, the debris detection equipment further acquires the background image corresponding to the target image stored in the equipment, and inputs both the target image and the background image into the debris detection model. The debris detection model inputs the target image to be reviewed into a first feature extraction module, and the background image into a second feature extraction module. The first and second feature extraction modules respectively extract features from the target image and the background image, obtaining a first feature map of the target image and a second feature map of the background image. These two feature maps are then input into a correlation acquisition module, which obtains a correlation feature image between the first feature map of the target image and the second feature map of the background image. The background image is a surveillance video frame image with the same scene as the target image, but without any debris.

[0075] Specifically, the first feature extraction module and the second feature extraction module are the backbone networks of classification networks such as residual neural networks and VGGNET deep neural networks. The first feature extraction module and the second feature extraction module can extract features from the input image, obtain the depth features of the image, and generate the corresponding feature image.

[0076] The first feature extraction module extracts features from the input target image, obtaining the first depth features of the target image and generating a first feature map of the target image. The second feature extraction module extracts features from the input background image, obtaining the second depth features of the background image and generating a second feature map. By using the first and second feature extraction modules to extract features from the target image and background image respectively, the influence of environmental factors such as lighting on the target image can be reduced, improving the accuracy of feature extraction. This, in turn, improves the accuracy of debris detection.

[0077] After acquiring the first feature map and the second feature map, the residue detection model inputs the first feature map and the second feature map into the correlation acquisition module. The correlation acquisition module compares the correlation between the first feature map and the second feature map to obtain the correlation feature images of the first feature map and the second feature map.

[0078] 203. Perform residue detection on the relevant feature images to obtain the residue detection results of the target image.

[0079] The debris detection model in the debris detection device compares the correlation between the first and second feature maps using a correlation acquisition module to obtain a relevant feature image. This relevant feature image is then input into the detection head module, which performs debris detection on the relevant feature image to obtain the debris detection result. The detection head module includes a convolution operation unit, a debris classification unit, and a debris localization unit. In one specific embodiment, the convolution operation unit performs two convolution stacks and has 96 channels. Optionally, the debris detection device can further adjust the convolution stack count and number of channels of the convolution operation unit according to the actual application scenario and computing resources.

[0080] Specifically, after acquiring relevant feature images, the detection head module inputs the relevant feature images into the convolution operation unit to perform convolution operations, obtains the convolution operation results, and inputs the convolution operation results into the waste classification unit and the waste location unit respectively.

[0081] Specifically, after receiving the convolution operation result, the residue classification unit performs residue classification on the convolution operation result, determines whether the changed image features in the convolution operation result match the residue features, and obtains the residue classification result, which includes residue and non-residue.

[0082] Specifically, after obtaining the convolution operation result, the object localization unit locates the changing image features in the convolution operation result and generates an object detection box for the changing image features.

[0083] The debris detection model generates debris detection results from the debris classification and location units and the debris detection box. The results are then output to the display interface of the debris detection device to alert the user that debris exists in the area.

[0084] In this embodiment, the object detection device acquires a target image to be detected, and separately acquires a first feature map of the target image and a second feature map of the corresponding background image. Correlation detection is performed on the first and second feature maps to obtain related feature images. After acquiring the related feature images, object detection is performed on them to obtain the object detection result. This achieves stable and accurate object detection and location in any scene, improves the anti-interference capability of the object detection model, reduces the interference of environmental factors such as lighting on the object detection model, improves the accuracy of object detection, and reduces the probability of false detection.

[0085] like Figure 3 As shown, Figure 3 This is a flowchart illustrating one embodiment of the method for obtaining relevant feature images in the method for detecting remnants provided in this application.

[0086] Based on the above embodiments, in this application embodiment, a relevant feature image is obtained according to the first feature map of the target image and the second feature map of the background image corresponding to the target image. Specifically, it includes steps 301-303:

[0087] 301. Extract the first feature block from the first feature map and the second feature block from the second feature map corresponding to the first feature block;

[0088] 302. Perform convolution operation on each of the first feature blocks and the corresponding second feature blocks to obtain each relevant feature;

[0089] 303. Integrate the aforementioned correlation features to obtain a correlation feature image.

[0090] After obtaining a first feature map of the target image and a second feature map of the background image through the first and second feature extraction modules in the debris detection model, the debris detection device inputs the first and second feature maps into a correlation acquisition module. The correlation acquisition module then acquires each first feature block of the first feature map and each corresponding second feature block in the second feature map. Correlation comparisons are performed based on the first and second feature blocks to generate a related feature image. Here, a first feature block is an image block in the first feature map centered at a point in the first feature map, carrying the feature information of the first feature map. Optionally, in a specific embodiment, the length of the first feature block K = 2k + 1, and its size is K*K. A second feature block is an image block in the second feature map centered at a specific point in the second feature map, carrying the feature information of the second feature map. Optionally, in a specific embodiment, the length of the second feature block K = 2k + 1, and its size is K*K.

[0091] After acquiring each first feature block and second feature block, the correlation acquisition module pairs the first feature block and the second feature block according to their corresponding points. For example, if the center point of the first feature block is x1, then the second feature block paired with the first feature block is the second feature block with the center point x2. Here, x1 and x2 are the same point corresponding to each other in the first feature map and the second feature map.

[0092] After pairing the first and second feature blocks, the correlation acquisition module performs a correlation convolution operation on the paired first and second feature blocks. The formula for the correlation convolution operation is as follows:

[0093]

[0094] The correlation acquisition module performs convolution on the first feature block and the second feature block using the correlation convolution operation formula to obtain the correlation features between the first feature block and the second feature block. These correlation features are image features characterizing the correlation between the first feature library and the second feature block.

[0095] The correlation acquisition module performs correlation convolution operations on all paired first and second feature blocks to obtain the correlation features of all first and second feature blocks. Based on these correlation features, it integrates the features to generate a correlation feature image of the first and second feature maps. This generated correlation feature image is then input to the detection head module, which performs residue detection on the correlation feature image to obtain the residue detection result.

[0096] In this embodiment, the debris detection device acquires each first feature block of the first feature map and each second feature block of the second feature map through the correlation acquisition module in the debris detection model. It then pairs the first and second feature blocks, performs a correlation convolution operation on the paired blocks, compares their correlation, generates correlation features, and generates a correlation feature image of the first and second feature maps based on these features. The device then uses this correlation feature image to detect debris in the target image, obtaining the debris detection result and improving the accuracy of debris detection.

[0097] like Figure 4 As shown, Figure 4 This is a flowchart illustrating an embodiment of the residue detection method provided in this application for detecting residues in the relevant feature image to obtain the residue detection result of the target image.

[0098] Based on the above embodiments, in this application embodiment, the relevant feature images are input into the detection head module. After the detection head module performs convolution operations on the relevant feature images, the residue classification and detection are performed. Specifically, this includes steps 401-402:

[0099] 401. Perform a classification convolution operation on the low-dimensional convolution result to obtain the change image features of the related feature image;

[0100] 402. Perform residue detection on the changed image features to obtain residue detection results, wherein the residue detection results include residues and non-residues.

[0101] The correlation acquisition module in the debris detection model generates relevant feature images and then inputs these images into the detection head module to detect debris, thus obtaining the debris detection results.

[0102] Specifically, the correlation acquisition module inputs the relevant feature image into the convolution operation unit of the detection head module. The convolution operation unit performs image upscaling on the relevant feature image, expanding the low-dimensional relevant feature image into a high-dimensional one. This high-dimensional relevant feature image is then input into the depthwise separable convolution sub-unit within the convolution operation unit for convolution, thus obtaining the high-dimensional convolution result. After obtaining the high-dimensional convolution result, a linear convolution mapping is used to map it back to a low-dimensional convolution result. This low-dimensional convolution result is then input into the residue classification unit for residue classification and detection, obtaining the residue detection result from the relevant feature image. This allows the residue detection model to reduce the number of computational parameters, compress the model size, and improve operational efficiency while maintaining performance.

[0103] After obtaining the result of the low-dimensional convolution operation, the residue classification unit performs a classification convolution operation on the result of the low-dimensional convolution operation to extract the change image features in the relevant feature images. The change image features are the difference image features between the target image and the background image.

[0104] After acquiring the changed image features, the residue classification unit performs residue detection on the changed image features to determine whether the changed image features are residues, thus obtaining the residue detection result. This residue detection result includes both residues and non-residues.

[0105] Specifically, the residue classification unit performs image recognition on the changed image features. If the image recognition result of the changed image feature matches the preset non-residue recognition result, then the changed image feature is determined to be a non-residue. Non-residues include substances that can leave traces, such as water, paint, and shadows.

[0106] If the image recognition result of the changed image feature does not match the preset non-relic recognition result, the changed image feature is determined to be a relic. The relic classification unit can further classify and identify the relic to determine the specific item type of the relic.

[0107] In this embodiment, the debris detection device performs dimensionality upscaling, convolution operations, and linear convolutional mapping on relevant feature images to obtain low-dimensional convolution results. These low-dimensional convolution results are then input into a debris classification unit, which classifies and detects the debris based on the low-dimensional convolution results, thus obtaining the debris detection result. This reduces the computational load of the debris detection model, improves operational efficiency, and enhances the accuracy of debris detection.

[0108] like Figure 5 As shown, Figure 5 This is a flowchart illustrating one embodiment of the method for locating debris in the debris detection method provided in this application.

[0109] Based on the above embodiments, in this application embodiment, after obtaining the low-dimensional convolution operation result, the residual object is located in the low-dimensional convolution operation result, specifically including steps 501-503:

[0110] 501. Perform a localization convolution operation on the low-dimensional convolution result to obtain the change image features of the relevant feature image;

[0111] 502. Obtain the object detection box of the changed image features, and obtain the position information of the object detection box;

[0112] 503. Locate the remaining objects based on the location information of the changed image features.

[0113] After obtaining the low-dimensional convolution operation result, the convolution operation unit inputs the low-dimensional convolution operation result into the legacy object localization unit, and the legacy object localization unit performs legacy object localization based on the low-dimensional convolution operation result.

[0114] Specifically, after receiving the low-dimensional convolution operation result, the debris localization unit performs a convolution operation on the low-dimensional convolution operation result to obtain the change image features of the relevant feature image. After obtaining the change image features, the debris localization unit performs regression processing on the change image features to obtain the debris detection box of the change image features, and obtains the position information of the debris detection box, wherein the position information of the debris detection box includes the center point, length and width of the debris detection box. Optionally, the shape of the debris detection box is a polygon.

[0115] Specifically, the debris localization unit performs regression processing on the changed image features to obtain the debris detection box. It then acquires the coordinates of each vertex of the detection box and calculates its length and width based on these coordinates. Simultaneously, it calculates the coordinates of the center point of the detection box based on the vertex coordinates. These center point coordinates, length, and width constitute the location information of the debris detection box. Based on this location information, the unit determines the location of the debris in the changed image features and outputs the location and classification result to the display interface of the debris detection device to remind the user to promptly address the debris.

[0116] In this embodiment, the abandoned object detection device performs abandoned object location detection on the results of low-dimensional convolution operations, determines the changing image features in the relevant feature images, and obtains the abandoned object detection box and corresponding location information of the changing image features. Based on the abandoned object detection box and corresponding location information, the device locates the abandoned object and outputs the abandoned object location result and abandoned object classification result to provide an abandoned object detection reminder. This allows users to quickly determine the type and location of the abandoned object, making it easier for users to retrieve the abandoned object in a timely manner.

[0117] To better implement the residue detection method in the embodiments of this application, based on the residue detection method, the embodiments of this application also provide a residue detection device, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of one embodiment of the residue detection device in this application. The residue detection device 600 includes:

[0118] Image acquisition module 601 is configured to acquire the target image to be detected;

[0119] The correlation detection module 602 is configured to obtain a correlation feature image based on a first feature map of the target image and a second feature map of the background image corresponding to the target image;

[0120] The residue detection module 603 is configured to perform residue detection on the relevant feature image to obtain the residue detection result of the target image.

[0121] In some embodiments of this application, the residue detection device 600 obtains a relevant feature image based on a first feature map of the target image and a second feature map of the background image corresponding to the target image, including:

[0122] The first feature map of the target image and the second feature map of the background image are input into the object detection model. The feature correlation between the first feature map and the second feature map is compared to generate a related feature image.

[0123] In some embodiments of this application, the residue detection device 600 obtains a relevant feature image based on a first feature map of the target image and a second feature map of the background image corresponding to the target image, including:

[0124] Extract the first feature block from the first feature map and the second feature block from the second feature map corresponding to the first feature block;

[0125] Perform convolution operations on each of the first feature blocks and the corresponding second feature blocks to obtain each relevant feature;

[0126] The aforementioned correlation features are integrated to obtain a correlation feature image.

[0127] In some embodiments of this application, after acquiring the target image to be detected, the object detection device 600 further includes:

[0128] The target image to be detected is input into the first feature extraction module for feature extraction to obtain the first feature map;

[0129] The background image corresponding to the target image is input into the second feature extraction module for feature extraction to obtain the second feature map.

[0130] In some embodiments of this application, the debris detection device 600 performs debris detection on the relevant feature image to obtain the debris detection result of the target image, including:

[0131] The relevant feature images are input into the object detection model, and the relevant feature images are subjected to image upscaling to obtain high-dimensional relevant feature images;

[0132] Perform convolution operation on the high-dimensional related feature image to obtain the high-dimensional convolution operation result;

[0133] The high-dimensional convolution operation result is mapped using a linear convolution machine to obtain the low-dimensional convolution operation result;

[0134] Based on the results of the low-dimensional convolution operation, the object classification and detection are performed to obtain the object detection results of the target image.

[0135] In some embodiments of this application, the debris detection device 600 performs debris classification and detection based on the low-dimensional convolution operation results to obtain debris detection results for the target image, including:

[0136] The low-dimensional convolution operation result is subjected to a classification convolution operation to obtain the change image features of the relevant feature image;

[0137] The changed image features are subjected to residue detection to obtain residue detection results, wherein the residue detection results include residues and non-residues.

[0138] In some embodiments of this application, before the debris detection device 600 performs debris detection on the relevant feature image and obtains the debris detection result of the target image, it further includes:

[0139] The low-dimensional convolution result is subjected to a localization convolution operation to obtain the change image features of the relevant feature image;

[0140] Obtain the object detection box of the changed image features, and obtain the position information of the object detection box;

[0141] Based on the location information, the features of the changed image are used to locate the remaining objects.

[0142] In some embodiments of this application, the debris detection device 600 acquires a debris detection box of the changed image features and acquires the position information of the debris detection box, including:

[0143] Obtain the coordinates of each vertex of the object detection frame, and calculate the length and width of the object detection frame based on the vertex coordinates;

[0144] The center point coordinates of the object detection frame are calculated based on the vertex coordinates, and the center point coordinates, length, and width of the object detection frame are determined as the position information of the object detection frame.

[0145] In this embodiment, the debris detection device acquires a target image to be detected, and separately acquires a first feature map of the target image and a second feature map of the corresponding background image. Correlation detection is performed on the first and second feature maps to obtain related feature images. After acquiring the related feature images, debris detection is performed on these related feature images to obtain the debris detection result. This achieves stable and accurate debris detection and localization in any scene, improves the anti-interference capability of the debris detection model, reduces the interference of environmental factors such as lighting on the debris detection model, improves the accuracy of debris detection, and reduces the probability of false detection.

[0146] This invention also provides a device for detecting abandoned items, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of one embodiment of the residue detection device provided in this application.

[0147] The abandoned object detection device integrates any of the abandoned object detection devices provided in the embodiments of the present invention, and the abandoned object detection device includes:

[0148] One or more processors;

[0149] Memory; and

[0150] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in the steps of the residue detection method described in any of the embodiments of the residue detection method described above.

[0151] Specifically, a debris detection device may include components such as a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a power supply 703, and an input unit 704. Those skilled in the art will understand that... Figure 7 The structure of the debris detection equipment shown does not constitute a limitation on the debris detection equipment. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0152] The processor 701 is the control center of the abandoned artifact detection device. It connects various parts of the device via interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702, thereby providing overall monitoring of the abandoned artifact detection device. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 701.

[0153] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the abandoned object detection device, etc. In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0154] The artifact detection equipment also includes a power supply 703 that supplies power to the various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 703 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0155] The artifact detection device may also include an input unit 704, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0156] Although not shown, the debris detection device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 701 in the debris detection device loads the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 runs the application programs stored in the memory 702 to realize various functions, as follows:

[0157] Acquire the target image to be detected;

[0158] Based on the first feature map of the target image and the second feature map of the corresponding background image, a relevant feature image is obtained;

[0159] The relevant feature images are subjected to residue detection to obtain the residue detection results of the target image.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0161] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0162] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0163] The above provides a detailed description of a method for detecting remnants provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting abandoned artifacts, characterized in that, The method for detecting the remains includes: Acquire the target image to be detected, and acquire a background image with the same image scene as the target image but without any remaining objects; Feature extraction is performed on the target image and the background image respectively to obtain a first feature map of the target image and a second feature map of the background image; a first feature block is extracted from the first feature map, and a second feature block corresponding to the first feature block is extracted from the second feature map, wherein the center point of the first feature block and the center point of the corresponding second feature block are the same point in the first feature map and the second feature map respectively; convolution operation is performed on each paired first feature block and the corresponding second feature block to obtain each correlation feature characterizing the correlation between the first feature block and the second feature block; the correlation features are integrated to obtain a correlation feature image; The relevant feature images are subjected to residue detection to obtain the residue detection results of the target image.

2. The method for detecting remnants as described in claim 1, characterized in that, The method for detecting abandoned items is applied to the abandoned item detection model; The step of obtaining a relevant feature image based on a first feature map of the target image and a second feature map of the corresponding background image includes: The first feature map of the target image and the second feature map of the background image are input into the object detection model. The feature correlation between the first feature map and the second feature map is compared to generate a related feature image.

3. The method for detecting remnants as described in claim 2, characterized in that, The artifact detection model also includes a first feature extraction module and a second feature extraction module; After acquiring the target image to be detected, the process further includes: The target image to be detected is input into the first feature extraction module for feature extraction to obtain the first feature map; The background image corresponding to the target image is input into the second feature extraction module for feature extraction to obtain the second feature map.

4. The method for detecting remnants as described in claim 2, characterized in that, The step of performing residue detection on the relevant feature images to obtain residue detection results for the target image includes: The relevant feature images are input into the object detection model, and the relevant feature images are subjected to image upscaling to obtain high-dimensional relevant feature images; Perform convolution operation on the high-dimensional related feature image to obtain the high-dimensional convolution operation result; The high-dimensional convolution operation result is mapped using a linear convolution machine to obtain the low-dimensional convolution operation result; Based on the results of the low-dimensional convolution operation, the object classification and detection are performed to obtain the object detection results of the target image.

5. The method for detecting remnants as described in claim 4, characterized in that, The step of classifying and detecting debris based on the low-dimensional convolution operation result to obtain the debris detection result of the target image includes: The low-dimensional convolution operation result is subjected to a classification convolution operation to obtain the change image features of the relevant feature image; The changed image features are subjected to residue detection to obtain residue detection results, wherein the residue detection results include residues and non-residues.

6. The method for detecting remnants as described in claim 4, characterized in that, Before performing residue detection on the relevant feature image to obtain the residue detection result of the target image, the method further includes: The low-dimensional convolution result is subjected to a localization convolution operation to obtain the change image features of the relevant feature image; Obtain the object detection box of the changed image features, and obtain the position information of the object detection box; Based on the location information, the features of the changed image are used to locate the remaining objects.

7. The method for detecting remnants as described in claim 6, characterized in that, The step of obtaining the object detection box for the changed image features and obtaining the position information of the object detection box includes: Obtain the coordinates of each vertex of the object detection frame, and calculate the length and width of the object detection frame based on the vertex coordinates; The center point coordinates of the object detection frame are calculated based on the vertex coordinates, and the center point coordinates, length, and width of the object detection frame are determined as the position information of the object detection frame.

8. A device for detecting abandoned items, characterized in that, The abandoned object detection device includes: The image acquisition module is configured to acquire the target image to be detected and to acquire a background image with the same image scene as the target image and without any remaining objects. The correlation detection module is configured to extract features from the target image and the background image respectively, to obtain a first feature map of the target image and a second feature map of the background image; extract a first feature block from the first feature map and a second feature block corresponding to the first feature block from the second feature map, wherein the center point of the first feature block and the center point of the corresponding second feature block are the same point in the first feature map and the second feature map; perform convolution operation on each paired first feature block and the corresponding second feature block to obtain each correlation feature characterizing the correlation between the first feature block and the second feature block; and integrate each correlation feature to obtain a correlation feature image. The residue detection module is configured to perform residue detection on the relevant feature image to obtain the residue detection result of the target image.

9. A device for detecting abandoned items, characterized in that, The abandoned artifact detection equipment includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the residue detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the residue detection method as described in any one of claims 1 to 7.

Images