Method and device for monitoring state of delivery, electronic device and storage medium

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for monitoring the status of disposed items. The method includes: based on a trained target detection model, performing abnormal disposal detection on the target image of the area to be identified at the current detection time to determine abnormal disposed items in the area to be identified at the current detection time; matching the abnormal disposed items at the current detection time with a historical disposal database, and updating the existence status of all abnormal disposed items in the historical disposal database in the area to be identified based on the matching results; wherein the historical disposal database includes abnormal disposed items detected in the area to be identified at all historical detection times. This method enables real-time monitoring of the existence status of disposed items, thereby achieving timely location and tracking of incorrectly disposed items, avoiding the impact of environmental interference and human obstruction on the accuracy of disposal detection, and ultimately improving the detection accuracy of waste sorting and disposal.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods, apparatus, electronic devices, and storage media for monitoring the status of objects being placed. Background Technology

[0002] Sorting waste according to function and type and placing it in the corresponding bins is a crucial step in the pre-treatment of waste. In practice, due to the large volume and high frequency of household waste disposal, it is necessary to control waste disposal to avoid the impact of mixed disposal on subsequent waste treatment.

[0003] Currently, artificial intelligence technology is commonly used to monitor and issue early warnings about waste disposal. Specifically, these technologies often identify non-kitchen waste types based on boundary information, or detect the hands and garbage bags of waste disposal personnel to determine whether disposal is standardized. However, current waste disposal monitoring is prone to misjudgments due to environmental interference and human obstruction, resulting in low accuracy in waste disposal detection.

[0004] There is currently no effective solution to the problem of low accuracy in waste disposal detection in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, electronic device, and storage medium for monitoring the status of disposed items, in order to solve the problem of low accuracy in waste disposal detection in related technologies.

[0006] Firstly, this embodiment provides a method for monitoring the status of deployed objects, including:

[0007] Based on the trained target detection model, abnormal placement detection is performed on the target image of the region to be identified at the current detection time to determine the abnormal placement objects in the region to be identified at the current detection time.

[0008] The abnormal delivery object at the current detection time is matched with the historical delivery database, and the existence status of all abnormal delivery objects in the historical delivery database in the area to be identified is updated based on the matching result; wherein, the historical delivery database includes abnormal delivery objects detected in the area to be identified at all historical detection times.

[0009] In some embodiments, the step of performing target matching between the anomalous delivery at the current detection time and the historical delivery database, and updating the existence status of all anomalous delivery items in the historical delivery database in the area to be identified based on the matching results, includes:

[0010] Based on image stitching features, the correlation between the abnormal delivery at the current detection time and other abnormal delivery in the historical delivery database is calculated;

[0011] The existence status of all abnormal deployments in the historical deployment database in the area to be identified is updated at least according to the correlation.

[0012] In some embodiments, the step of performing target matching between the abnormal delivery at the current detection time and the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, further includes:

[0013] Abnormal delivery items that do not match the abnormal delivery items at the current detection time are filtered out from the historical delivery database and identified as non-visible delivery items;

[0014] The trained state recognition model is used to identify the state of the non-visible object to obtain a quasi-state result.

[0015] Based on the quasi-state results and the detection characteristics of the non-visible object, the existence status of the non-visible object is identified as either removed or covered.

[0016] In some embodiments, the step of using a trained state recognition model to perform state recognition on the non-visible object to obtain a quasi-state result includes:

[0017] The historical image sequence of a preset number of frames between the current detection time and the previous detection time is input into the trained state recognition model to perform state recognition on the non-visible object and obtain the quasi-state result.

[0018] In some embodiments, identifying the existence state of the non-visible object as either removed or covered based on the quasi-state result and the detection characteristics of the non-visible object includes:

[0019] The quasi-state result is spliced ​​with the detection features of the non-visible object detected at a preset historical detection time, and the splicing result is input into a preset multilayer perceptron for classification of the existence state, determining whether the existence state of the non-visible object is a removed state or a covered state.

[0020] In some embodiments, the step of performing target matching between the abnormal delivery at the current detection time and the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, further includes:

[0021] The abnormal delivery items at the current detection time are matched with the historical delivery database.

[0022] In response to the existence of an abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time, the existence status of the abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time is identified as an unprocessed state.

[0023] If no abnormal delivery item is found in the historical delivery database that matches the abnormal delivery item at the current detection time, the abnormal delivery item at the current detection time is added to the historical delivery database.

[0024] In some embodiments, the method further includes:

[0025] According to a preset order, alarm information for all abnormal deployments in the historical deployment database is output sequentially.

[0026] Secondly, this embodiment provides a status monitoring device for deployed objects, including: a detection module and a matching module; wherein:

[0027] The detection module is used to perform abnormal delivery detection on the target image of the region to be identified at the current detection time based on the trained target detection model, and to determine the abnormal delivery object of the region to be identified at the current detection time.

[0028] The matching module is used to perform target matching between the abnormal delivery at the current detection time and the historical delivery database, and to update the existence status of all abnormal delivery in the historical delivery database in the area to be identified based on the matching result; wherein, the historical delivery database includes abnormal delivery detected in the area to be identified at all historical detection times.

[0029] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the status monitoring method for the deployed object described in the first aspect above.

[0030] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the status monitoring method for the deployed object described in the first aspect.

[0031] Compared with related technologies, the method, apparatus, electronic device, and storage medium for monitoring the status of disposed items provided in this embodiment, based on a trained target detection model, performs abnormal disposal detection on the target image of the area to be identified at the current detection time, and determines the abnormal disposed items in the area to be identified at the current detection time; it matches the abnormal disposed items at the current detection time with the historical disposal database, and updates the existence status of all abnormal disposed items in the area to be identified based on the matching results; wherein, the historical disposal database includes abnormal disposed items detected in the area to be identified at all historical detection times. It can achieve real-time monitoring of the existence status of disposed items, thereby enabling timely location and tracking of incorrectly disposed items, avoiding the impact of environmental interference and human obstruction on the accuracy of disposal detection, and thus improving the detection accuracy of waste sorting and disposal.

[0032] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a hardware structure block diagram of the terminal of the method for monitoring the status of the deployed objects in this embodiment;

[0035] Figure 2 This is a flowchart of the method for monitoring the status of the deployed items in this embodiment;

[0036] Figure 3 This is a schematic diagram of a waste disposal monitoring scenario in this embodiment;

[0037] Figure 4 This is a schematic diagram of the state recognition model in this embodiment;

[0038] Figure 5 This is a flowchart of the waste mixed disposal monitoring method according to a preferred embodiment;

[0039] Figure 6 This is a structural block diagram of the status monitoring device for the deployed items in this embodiment. Detailed Implementation

[0040] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0041] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0042] The method embodiments provided in this example can be executed on a terminal, such as a computer or smart camera, or on a server. For example, after the camera captures images, it uploads them to the server, where the server can then recognize them. Alternatively, it can be executed in the cloud or on a distributed system. Taking execution on a terminal as an example... Figure 1 This is a hardware structure block diagram of the terminal for the status monitoring method of the deployed object in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0043] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for monitoring the status of the released object in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0045] This embodiment provides a method for monitoring the status of deployed materials. Figure 2 This is a flowchart of the status monitoring method for the deployed items in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0046] Step S210: Based on the trained target detection model, perform abnormal delivery detection on the target image of the region to be identified at the current detection time to determine the abnormal delivery objects in the region to be identified at the current detection time.

[0047] This target detection model refers to a model trained based on a specified type of object being placed, which can be implemented using any image detection algorithm. The aforementioned region to be identified refers to the area where abnormal object detection is required. The aforementioned target image is obtained by capturing images of the region to be identified within the application scenario. Furthermore, the abnormal objects in this embodiment are those restricted from appearing in the region to be identified according to relevant regulations in the actual application scenario. Additionally, to avoid interference from environmental factors and human occlusion during the placement process, this embodiment directly monitors the placement results in the region to be identified rather than the placement behavior, thereby avoiding misjudgments in placement detection.

[0048] The following explanation uses a waste sorting and disposal monitoring scenario as an example. Monitoring equipment can be deployed above the waste bins being monitored, or in any other location that covers the interior area of ​​the waste bins from a specific angle. Figure 3 This is a schematic diagram illustrating a waste disposal monitoring scenario in this embodiment. Figure 3 As shown, if the trash can allows the disposal of kitchen waste, the types of waste that should not be disposed of in the trash can can be pre-configured. For example, incorrectly disposed waste types include aluminum cans, plastic bottles, cardboard boxes, and plastic bags. Furthermore, the aforementioned object detection model can be a trained model used to detect everyday waste and household items. The monitoring equipment acquires an image of the area inside the trash can at the current detection time, i.e., an image of the waste disposed of at the current detection time, and inputs it into the trained object detection model for abnormal disposal detection. Figure 3 As shown by the solid black box in the image, when the object detection model detects incorrectly placed plastic bottles and bags in the target image, it will output the abnormal object. For example, the output information of the abnormal object may include the type of object and the coordinates of its bounding rectangle in the target image.

[0049] Step S220: Match the abnormal delivery at the current detection time with the historical delivery database, and update the existence status of all abnormal delivery in the area to be identified based on the matching results; wherein, the historical delivery database includes abnormal delivery detected in the area to be identified at all historical detection times.

[0050] Within the monitoring period, a detection time can be set at fixed time intervals, such as 2 seconds. Images captured by the monitoring device in the area to be identified are periodically input into the target detection model, generating a set of abnormal objects for each corresponding time. Therefore, storing all abnormal objects detected within the preset time period into a preset processing pool yields the historical deployment library of this embodiment. That is, the historical deployment library in this embodiment refers to a collection of information on all abnormal deployments from the start of image acquisition up to the previous detection time. The aforementioned historical detection times are the detection times before the current detection time when abnormal deployment detection was performed on the area to be identified. This historical deployment library can be located in a cache or stored in a preset database. The specific storage method and storage medium can be adaptively designed according to actual application requirements; this embodiment does not impose specific limitations on this.

[0051] Furthermore, the persistence status of abnormal delivery items in the area to be identified can include states such as removed, covered, or unprocessed. Understandably, as new delivery items are added and human intervention occurs, the persistence status of abnormal delivery items in the area to be identified will change. For example, an abnormal delivery item detected at a certain detection time may be manually removed or covered by other delivery items before a new detection time arrives. Based on this, in order to track the persistence status of abnormal delivery items already existing in the historical delivery database, this embodiment performs target matching between the abnormal delivery items at the current detection time and the historical delivery database, thereby determining the persistence status of all abnormal delivery items in the historical delivery database.

[0052] Specifically, the anomalous delivery at the current detection time can be correlated and matched with anomalous delivery items detected at all historical detection times in the historical delivery database to determine whether there are anomalous delivery items in the historical delivery database that belong to the same delivery item as the currently detected anomalous delivery item. The specific target matching method can be achieved by calculating the similarity of image features. For example, image features such as color, texture, contour, key points, spatial distribution, and positional features can be referenced to calculate the similarity between the anomalous delivery item at the current detection time and other anomalous delivery items already existing in the historical delivery database. Preferably, this embodiment combines positional features, color features, and texture features to calculate the similarity to obtain the target matching result between the anomalous delivery item at the current detection time and the historical delivery database, thereby improving the accuracy of target matching.

[0053] Compared to related technologies that only identify the delivery action or rely solely on single boundary information, resulting in low accuracy in anomaly detection, this embodiment tracks the continued existence of all detected anomaly items within the target area. This continuous monitoring of the anomaly items' subsequent states improves detection accuracy. Furthermore, this embodiment allows for the specification of error delivery types based on specific application scenarios, thus adapting to anomaly detection in various contexts.

[0054] Optionally, if an abnormal delivery item that matches the abnormal delivery item at the current detection time exists in the historical delivery database, meaning the abnormal delivery item at the current detection time already exists in the historical delivery database, it indicates that the abnormal delivery item has not been cleaned up. The existence status of the abnormal delivery item that matches the abnormal delivery item at the current detection time in the historical delivery database can be identified as unprocessed. Conversely, if no abnormal delivery item matches the abnormal delivery item at the current detection time in the historical delivery database, the abnormal delivery item at the current detection time can be added to the historical delivery database, and its existence status can be identified as unprocessed. It is worth noting that there can be one or more abnormal delivery items at the current detection time. When there are multiple abnormal delivery items at the current detection time, each abnormal delivery item needs to be matched against the historical delivery database separately, and the aforementioned existence status needs to be set based on the matching result of each abnormal delivery item. For example, if the set of abnormal delivery items obtained at the current detection time t is S... t The historical deployment database contains a set S of all anomalous deployments up to time t-1. t-1 The above target matching refers to matching S... t All anomalous deliverables and S t-1 The similarity of all abnormal delivery items is calculated to update the historical delivery database.

[0055] Specifically, if there are abnormal delivery items in the historical delivery database that do not match the abnormal delivery items at the current detection time, it indicates that there are abnormal delivery items in the historical delivery database that were not detected at the current detection time. This suggests that such abnormal delivery items may have been removed or obscured by other delivery items, and these abnormal delivery items can be identified as invisible delivery items. Therefore, in this case, this embodiment determines the accurate existence status of invisible delivery items in the historical delivery database by performing state recognition. Specifically, a state recognition model can be constructed based on a long short-term memory structure to process historical image sequences within a preset time interval, and the detection information of invisible delivery items detected at historical detection times can be fused to determine their existence status as either removed or covered. The preset time interval can be the interval formed from the last historical detection time when the invisible delivery item was detected to the current detection time when it was not detected. Compared with related technologies, which cannot handle the tracking of abnormal delivery item states in scenarios such as delivery item stacking and obscuration, thus leading to misjudgments in abnormal delivery monitoring and low accuracy of detection results, this approach addresses the problem. This embodiment continuously updates the existence status of abnormal objects in the area to be identified, enabling accurate detection even when objects are obscured or covered. This avoids missing abnormal objects, prevents misjudgments in delivery monitoring, and improves the accuracy of abnormal delivery detection.

[0056] Steps S210 to S220 above are omitted. Based on the trained target detection model, abnormal placement detection is performed on the target image of the area to be identified at the current detection time to determine the abnormal placement items in the area to be identified at the current detection time. The abnormal placement items at the current detection time are matched with the historical placement database, and the survival status of all abnormal placement items in the historical placement database in the area to be identified is updated based on the matching results. The historical placement database includes abnormal placement items detected in the area to be identified at all historical detection times. This enables real-time monitoring of the survival status of placed items, thereby achieving timely location and tracking of incorrectly placed items, avoiding the impact of environmental interference and human occlusion on the accuracy of placement detection, and thus improving the detection accuracy of waste sorting and placement.

[0057] Furthermore, in one embodiment, based on the above step S220, the abnormal delivery at the current detection time is matched with the historical delivery database, and the existence status of all abnormal delivery items in the historical delivery database in the area to be identified is updated based on the matching results. Specifically, this may include the following steps:

[0058] Step S221: Based on image stitching features, calculate the correlation between the abnormal delivery at the current detection time and other abnormal delivery in the historical delivery database.

[0059] This image stitching feature refers to the feature formed by stitching together multiple extracted image features. For example, the color histogram feature within the detection box corresponding to the anomalous object can be stitched together with the scale-invariant feature transform (SIFT) to obtain the image stitching feature. Specifically, the set S of all anomalous objects at the current detection time... t In the image stitching feature corresponding to the i-th anomalous delivery item, the following is... The set S of all abnormal deployments in the historical deployment database up to the detection time t-1 t-1 The image stitching feature of the j-th anomalous object is: S one by one t Each anomalous delivery item in S t-1 The correlation degree between each anomalous delivery item is calculated. Specifically, cosine similarity can be calculated, and the Intersection over Union (IoU) can be used as a position coefficient to obtain the correlation degree C between two anomalous delivery items. ij The specific calculation method is shown in the following formula:

[0060]

[0061] Among them, C ij S representst The i-th abnormal item and S t-1 The correlation between the j-th anomalous delivery items. IoU ij S represents t The i-th abnormal item and S t-1 The overlap between the j-th abnormal delivery items.

[0062] Next, the correlation C was selected. ij Association pairs exceeding the threshold will be S t The abnormal delivery items are matched with the abnormal delivery items with the highest correlation Cij. If the match is successful, it means that the abnormal delivery item existed at the previous detection time and is still in an unprocessed state; if the match fails, it means that the abnormal delivery item is a newly added delivery item, and its location, image information, and first appearance time can be added to the historical delivery database.

[0063] Step S222: Update the existence status of all abnormal delivery items in the area to be identified in the historical delivery database at least according to the correlation.

[0064] Steps S221 to S222 above, based on image stitching features, obtain the correlation between the abnormal delivery at the current detection time and other abnormal delivery in the historical delivery database, thereby improving the accuracy of target matching.

[0065] In another embodiment, based on the above step S220, the abnormal delivery at the current detection time is matched with the historical delivery database, and the existence status of all abnormal delivery items in the area to be identified is updated based on the matching results. This may further include the following steps:

[0066] Step S223: Filter out anomalous delivery objects from the historical delivery database that do not match the anomalous delivery objects at the current detection time, and identify them as invisible delivery objects. That is, invisible delivery objects refer to anomalous delivery objects that existed in the area to be identified at the previous detection time but were not detected at the current detection time. The existence status of such anomalous delivery objects includes two types: a removed state (having been removed from the area to be identified) and a covered state (being covered by other delivery objects or occluded by other things).

[0067] Step S224: Use the trained state recognition model to identify the state of the non-visible object and obtain the quasi-state result.

[0068] Specifically, this embodiment utilizes the trained state recognition model to perform probability prediction and cross-entropy classification loss calculation on the image sequence of the region to be identified within a preset time interval for the aforementioned two existence states, thereby obtaining quasi-state results. The aforementioned state recognition model can specifically be a hybrid neural network model based on a combination of convolutional neural networks and long short-term memory networks.

[0069] Step S225: Based on the quasi-state results and the detection characteristics of non-visible objects, the existence status of non-visible objects is identified as either removed or covered.

[0070] Specifically, based on the quasi-state result obtained in step S224 above, and combined with the detection characteristics of the non-visible object detected at historical detection times, a multilayer perceptron is used for classification, thereby determining the existence state of the non-visible object from the removed state and the covered state. Steps S223 to S225 above identify the removed and covered states of the non-visible object based on the state recognition model, thus accurately identifying the existence state of obscured abnormal objects. This overcomes the problems of misjudgment and missed detection caused by obstruction interference factors in related technologies, and also avoids the omission of mixed waste disposal detection. Therefore, the state monitoring of objects provided in this embodiment can improve the accuracy of abnormal disposal detection results.

[0071] Furthermore, in one embodiment, based on the above step S224, the trained state recognition model is used to perform state recognition on the non-visible object to obtain a quasi-state result. Specifically, this may include: inputting a historical image sequence of a preset number of frames between the current detection time and the previous detection time into the trained state recognition model to perform state recognition on the non-visible object and obtain a quasi-state result.

[0072] The state recognition model is a hybrid neural network model combining a convolutional neural network and a long short-term memory (LSTM) network. Specifically, a predetermined number of historical image frames are extracted from the previous detection time t-1 to the current detection time t. This historical image sequence consists of images acquired by the monitoring device from time t-1 to time t, covering the area to be identified. More specifically, the historical image sequence can be extracted from the detection time when the non-visible object last appeared to the current detection time when it disappeared.

[0073] Figure 4 This is a schematic diagram of the state recognition model in this embodiment. Figure 4As shown, for each frame of the historical image sequence, it needs to be input into a multi-layer convolutional neural network before being output to the corresponding LSTM unit. Specifically, the first frame of the historical image is input into the first LSTM unit after being processed by the multi-layer convolutional neural network. The first LSTM unit then outputs the processed information to the second LSTM unit. The second LSTM unit processes the information from the first LSTM unit, as well as the information from the currently received second frame of the historical image after processing by the multi-layer convolutional neural network. This process continues, with the i-th frame of the historical image having its image features extracted by the multi-layer convolutional neural network and then input into the i-th LSTM unit. The i-th LSTM unit processes the information produced by the (i-1)-th LSTM unit and the extracted image features from the currently received i-th frame of the historical image, and then inputs it into the (i+1)-th LSTM unit. The output of the last LSTM unit is the aforementioned quasi-state result, which includes the probability that the non-visible object belongs to the removed state or the covered state.

[0074] This embodiment constructs a state recognition model based on a long short-term memory structure, using historical image frames within the time period during which the non-visible object disappears from the area to be identified as input, thereby enabling accurate prediction of the existence status of the non-visible object.

[0075] Furthermore, in one embodiment, based on the above step S225, the existence status of the non-visible object is identified as either removed or covered according to the quasi-state result and the detection features of the non-visible object. Specifically, this may include: splicing the quasi-state result and the detection features of the non-visible object detected at a preset historical detection time, and inputting the splicing result into a preset multilayer perceptron for classification of the existence status, thereby determining that the existence status of the non-visible object is either removed or covered.

[0076] Continue to combine Figure 4 The following explanation is provided. After the state recognition model outputs the quasi-state result, the detection features of the non-visible object detected by the target detection model at historical detection times are extracted. These features include, for example, the coordinates of the bounding rectangle and the object type. The quasi-state result is then concatenated with these detection features and input into a multilayer perceptron for processing, thereby obtaining the classification result of the non-visible object between the removed and covered states. Preferably, the detection features can be those obtained when the non-visible object is first detected.

[0077] When the existence status of a non-visible waste item is determined to be "removed," it can be removed from the historical waste disposal database. This embodiment, by fusing detection features with the aforementioned quasi-state results, can obtain the accurate existence status of non-visible waste items, thereby enabling the detection of whether abnormally disposed waste has been cleared in scenarios of waste accumulation, and thus improving the accuracy and comprehensiveness of detection results in abnormal waste disposal scenarios.

[0078] In another embodiment, based on the above step S220, the abnormal delivery at the current detection time is matched with the historical delivery database, and the existence status of all abnormal delivery items in the area to be identified is updated based on the matching results. This may further include the following steps:

[0079] Step S226: Match the abnormal delivery items at the current detection time with the historical delivery database;

[0080] Step S227: In response to the existence of an abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time, the existence status of the abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time is identified as an unprocessed state.

[0081] Step S228: In response to the absence of an abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time, the abnormal delivery item at the current detection time is added to the historical delivery database. Steps S226 to S228 above enable real-time updates to the historical delivery database, thereby achieving continuous monitoring of abnormal delivery items in the area to be identified.

[0082] In another embodiment, the above-described method for monitoring the status of the deployed object may further include the following steps:

[0083] Step S230: Output alarm information for all abnormal deployments in the historical deployment database in a preset order.

[0084] Specifically, the existence time of each abnormal item in the historical deployment database is calculated, sorted, and then stored sequentially in a preset alarm queue. For example, abnormal items are stored in the alarm queue in descending order of their existence time. Based on this alarm queue, the image, time information, and other details of the abnormal item are input to trigger an alarm, prompting relevant personnel to take appropriate measures. This embodiment can continuously monitor the dynamics of abnormal items in the alarm queue and trigger alarms, thereby facilitating deployment management.

[0085] The present embodiment will now be described and illustrated through preferred embodiments.

[0086] Figure 5 This is a flowchart of the waste mixing monitoring method according to a preferred embodiment. Figure 5 As shown, this waste mixing monitoring method includes the following steps:

[0087] Step S501, abnormal object detection, to obtain the abnormal objects at the current detection time; wherein, based on the trained target detection model, the abnormal objects that are incorrectly placed in the area to be identified are detected to obtain the abnormal objects;

[0088] Step S502: Associate the currently detected abnormal delivery items with the historical delivery database;

[0089] Step S503: Determine whether there is an abnormal delivery item in the historical delivery database that is associated with the currently detected abnormal delivery item; if so, proceed to step S504; otherwise, proceed to step S505.

[0090] Step S504: Maintain the unprocessed state of the abnormally delivered item;

[0091] Step S505: Add the currently detected abnormal delivery items to the historical delivery database;

[0092] Step S506: Generate an alarm queue;

[0093] Step S507: Filter out non-visible delivery items that disappeared at the current detection time from the historical delivery database;

[0094] Step S508: Input the filtering results from step S507 into the state recognition model for processing to obtain the state recognition results;

[0095] Step S509: Determine whether the non-visible object has been removed based on the status recognition result; if yes, proceed to step S510; otherwise, proceed to step S511.

[0096] Step S510: Remove the removed non-visible deliverables from the historical deliverables library;

[0097] Step S511: Identify the state of the non-visible object as covered and perform anomaly handling.

[0098] This embodiment also provides a status monitoring device for the deployed object, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 6 This is a structural block diagram of the status monitoring device 60 for the released items in this embodiment, as shown below. Figure 6As shown, the status monitoring device 60 for the deployed object includes: a detection module 62 and a matching module 64; wherein: the detection module 62 is used to perform abnormal deployment detection on the target image of the region to be identified at the current detection time based on the trained target detection model, and determine the abnormal deployed object in the region to be identified at the current detection time; the matching module 64 is used to perform target matching between the abnormal deployed object at the current detection time and the historical deployment database, and update the existence status of all abnormal deployed objects in the region to be identified based on the matching results; wherein, the historical deployment database includes abnormal deployed objects detected in the region to be identified at all historical detection times.

[0100] The aforementioned status monitoring device 60 can monitor the status of the disposed items in real time, thereby enabling timely location and tracking of incorrectly disposed items, avoiding the impact of environmental interference and human obstruction on the accuracy of disposal detection, and thus improving the detection accuracy of waste sorting and disposal.

[0101] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0102] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0103] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0104] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0105] S1, Based on the trained target detection model, perform abnormal placement detection on the target image of the region to be identified at the current detection time, and determine the abnormal placement objects in the region to be identified at the current detection time;

[0106] S2, perform target matching between the abnormal delivery at the current detection time and the historical delivery database, and update the existence status of all abnormal delivery in the region to be identified based on the matching results; wherein, the historical delivery database includes abnormal delivery detected in the region to be identified at all historical detection times.

[0107] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0108] Furthermore, in conjunction with the status monitoring method for deployed objects provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the status monitoring methods for deployed objects in the above embodiments.

[0109] It should be noted that the information and data involved in this application (including but not limited to data used for analysis, stored data, and displayed data) comply with relevant laws and regulations.

[0110] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0111] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for monitoring the status of a deployed object, characterized in that, include: Based on the trained target detection model, abnormal placement detection is performed on the target image of the region to be identified at the current detection time to determine the abnormal placement objects in the region to be identified at the current detection time. The abnormal delivery at the current detection time is matched with the historical delivery database, and the existence status of all abnormal delivery items in the historical delivery database in the area to be identified is updated based on the matching results; wherein, the historical delivery database includes abnormal delivery items detected in the area to be identified at all historical detection times; The step of matching the abnormal delivery at the current detection time with the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, includes: Abnormal delivery items that do not match the abnormal delivery items at the current detection time are filtered out from the historical delivery database and identified as non-visible delivery items; The trained state recognition model is used to identify the state of the non-visible object to obtain a quasi-state result. Based on the quasi-state results and the detection characteristics of the non-visible object, the existence status of the non-visible object is identified as either removed or covered.

2. The method for monitoring the status of the deployed object according to claim 1, characterized in that, The step of matching the abnormal delivery at the current detection time with the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, further includes: Based on image stitching features, the correlation between the abnormal delivery at the current detection time and other abnormal delivery in the historical delivery database is calculated. The existence status of all abnormal deployments in the historical deployment database in the area to be identified is updated at least according to the correlation.

3. The method for monitoring the status of the deployed object according to claim 1, characterized in that, The step of using the trained state recognition model to perform state recognition on the non-visible object to obtain a quasi-state result includes: The historical image sequence of a preset number of frames between the current detection time and the previous detection time is input into the trained state recognition model to perform state recognition on the non-visible object and obtain the quasi-state result.

4. The method for monitoring the status of the deployed object according to claim 1, characterized in that, The step of identifying the existence state of the non-visible object as either removed or covered based on the quasi-state result and the detection characteristics of the non-visible object includes: The quasi-state result is spliced ​​with the detection features of the non-visible object detected at a preset historical detection time, and the splicing result is input into a preset multilayer perceptron for classification of the existence state, determining whether the existence state of the non-visible object is a removed state or a covered state.

5. The method for monitoring the status of the deployed object according to claim 1, characterized in that, The step of matching the abnormal delivery at the current detection time with the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, further includes: The abnormal delivery items at the current detection time are matched with the historical delivery database. In response to the existence of an abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time, the existence status of the abnormal delivery item in the historical delivery database that successfully matches the abnormal delivery item at the current detection time is identified as an unprocessed state. If no abnormal delivery item is found in the historical delivery database that matches the abnormal delivery item at the current detection time, the abnormal delivery item at the current detection time is added to the historical delivery database.

6. The method for monitoring the status of a deployed object according to any one of claims 1 to 5, characterized in that, The method further includes: According to a preset order, alarm information for all abnormal deployments in the historical deployment database is output sequentially.

7. A status monitoring device for a released object, characterized in that, include: Detection module and matching module; wherein: The detection module is used to perform abnormal delivery detection on the target image of the region to be identified at the current detection time based on the trained target detection model, and to determine the abnormal delivery object of the region to be identified at the current detection time. The matching module is used to perform target matching between the abnormal delivery at the current detection time and the historical delivery database, and to update the existence status of all abnormal delivery in the historical delivery database in the area to be identified based on the matching result; wherein, the historical delivery database includes abnormal delivery detected in the area to be identified at all historical detection times; The step of matching the abnormal delivery at the current detection time with the historical delivery database, and updating the existence status of all abnormal delivery items in the historical delivery database in the area to be identified based on the matching results, includes: Abnormal delivery items that do not match the abnormal delivery items at the current detection time are filtered out from the historical delivery database and identified as non-visible delivery items; The trained state recognition model is used to identify the state of the non-visible object to obtain a quasi-state result. Based on the quasi-state results and the detection characteristics of the non-visible object, the existence status of the non-visible object is identified as either removed or covered.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method for monitoring the status of the released object as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for monitoring the status of the deployed object as described in any one of claims 1 to 6.

Images