Target detection method and device and electronic equipment

Abstract

The invention discloses a target detection method and device and electronic equipment in the field of artificial intelligence, and the method comprises the steps: detecting N to-be-detected pictures through a target detection model, and obtaining a first danger detection result corresponding to each to-be-detected picture; based on the first danger detection result, extracting K to-be-detected pictures with danger information from the N to-be-detected pictures; and detecting the K to-be-detected pictures by using the multi-modal large model to obtain a second danger detection result corresponding to each to-be-detected picture, so that the N to-be-detected pictures are preliminarily screened, and then the screened K to-be-detected pictures are detected by using the picture understanding capability of the multi-modal large model. Therefore, the false alarm rate of the dangerous information in the to-be-detected picture is reduced, and the accuracy of whether the dangerous information exists in the to-be-detected picture is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to an object detection method, apparatus, and electronic device. Background Art

[0002] In various scenarios of video surveillance, the captured pictures can be detected to detect dangerous information in the pictures. For example, in the scenario of road video surveillance, the captured pictures can be detected to detect whether there is tripartite construction in the picture. If there is tripartite construction, it means that there is dangerous information in the picture, so the dangerous information in the picture is detected. Furthermore, after detecting the dangerous information, potential safety hazards can be eliminated in a timely and effective manner.

[0003] Currently, a single small model (such as the YOLO model) is usually used to detect pictures, and then whether there is dangerous information in the picture is determined according to the dangerous detection result of the picture, resulting in low detection accuracy of dangerous information in the picture. Summary of the Invention

[0004] This application provides an object detection method, apparatus, and electronic device to achieve the effect of improving the detection accuracy of dangerous information. The specific implementation solutions are as follows:

[0005] In a first aspect, this application provides an object detection method, and the method includes:

[0006] Using an object detection model, detect N pictures to be detected respectively, and obtain the first dangerous detection result corresponding to each picture to be detected; where N is a positive integer;

[0007] Based on the first dangerous detection result, extract K pictures to be detected with dangerous information from the N pictures to be detected; where K is a positive integer less than or equal to N;

[0008] Using a multimodal large model, detect the K pictures to be detected respectively, and obtain the second dangerous detection result corresponding to each picture to be detected.

[0009] Through the above application embodiments, the first risk detection results of N pictures to be detected are obtained by using the target detection model. K pictures to be detected with risk information are preliminarily screened out from the N pictures to be detected, thereby removing the pictures to be detected that definitely do not have risk information, so as to reduce the false alarm rate of the pictures to be detected with risk information. Then, based on the K pictures to be detected that are preliminarily determined to have risk information, a multi-modal large model is used to detect them, thereby reducing the number of picture detections of the multi-modal large model. Due to the architecture of the multi-modal large model, the multi-modal large model has excellent picture understanding ability. Therefore, by using the picture understanding ability of the multi-modal large model, it is possible to more accurately detect whether there is risk information in the pictures to be detected. Therefore, first, the first risk detection results of the target detection model are used to preliminarily screen the N pictures to be detected, and then the picture understanding ability of the multi-modal large model is used to detect the K pictures to be detected that are preliminarily determined to have risk information. Thus, accurate detection of the pictures to be detected is achieved through a multi-model approach, significantly reducing the false alarm rate of the pictures to be detected with risk information, improving the detection accuracy of whether there is risk information in the pictures to be detected, and at the same time ensuring the detection speed of the pictures to be detected, so that potential safety hazards can be eliminated more accurately and efficiently.

[0010] In a possible implementation manner, if the target detection model is a small target detection model, then based on the first risk detection result, extracting K pictures to be detected with risk information from the N pictures to be detected includes:

[0011] Extracting M pictures to be detected from the N pictures to be detected, where the first risk detection result indicates the existence of the risk information; where M is a positive integer less than or equal to N;

[0012] Using a multi-target detection model to detect each of the M pictures to be detected, and obtaining the corresponding third risk detection result for each picture to be detected;

[0013] Extracting the K pictures to be detected from the M pictures to be detected, where the third risk detection result indicates the existence of the risk information; where K is a positive integer less than or equal to M.

[0014] Through the above application embodiments, based on the first hazard detection results of the target detection small model, M to-be-detected pictures with hazard information were preliminarily screened out from N to-be-detected pictures. Then, the multi-object detection model was used to detect these M to-be-detected pictures, and further K to-be-detected pictures with hazard information were screened out. Thus, before extracting the K to-be-detected pictures, the to-be-detected pictures were screened multiple times, and further the to-be-detected pictures that definitely do not have hazard information were removed, so that the probability of the to-be-detected pictures without hazard information among the finally extracted K to-be-detected pictures is lower, which is convenient for multi-level filtering of the to-be-detected pictures. Thereby, the detection accuracy of the to-be-detected pictures with hazard information has been greatly improved, and at the same time, the detection speed is also ensured, and then potential safety hazards can be eliminated more accurately and efficiently.

[0015] In a possible implementation manner, using the multi-modal large model to detect each of the K to-be-detected pictures respectively, and obtaining the corresponding second hazard detection results for each to-be-detected picture, includes:

[0016] Based on the multi-modal large model including preset rules, respectively determine whether each of the K to-be-detected pictures conforms to the preset rules;

[0017] If so, generate the second hazard detection result corresponding to the to-be-detected picture, including the existence of the hazard information;

[0018] If not, generate the second hazard detection result corresponding to the to-be-detected picture, including the non-existence of the hazard information.

[0019] Through the above application embodiments, using the multi-modal large model including preset rules, it is determined whether each of the K to-be-detected pictures conforms to the preset rules. Thus, based on the judgment results, it is accurately determined whether there is hazard information in the corresponding to-be-detected pictures.

[0020] In a possible implementation manner, the preset rule is that there is any one of an orange retractable fence, a red retractable fence, a fence of any color, and a construction fence in the to-be-detected picture;

[0021] Or, the preset rule is that there is construction machinery in the corresponding to-be-detected picture;

[0022] Or, the preset rule is that there are personnel wearing safety helmets in the corresponding to-be-detected picture.

[0023] Based on the above preset rules, which are for the orange retractable fence, red retractable fence, any - colored enclosure, construction enclosure, construction machinery, and personnel wearing safety helmets, it is possible to accurately determine whether there is content in the to - be - detected picture that conforms to the characteristics of dangerous information (such as tri - party construction), thereby accurately determining whether there is dangerous information in the to - be - detected picture, and further more accurately eliminating potential safety hazards in the scenario of dangerous information.

[0024] In a possible implementation manner, the target detection model is a small target detection model or a multi - target detection model, and the method further includes:

[0025] Using the multi - modal large model, process the to - be - detected picture, the detection box indicating the positional relationship between the ladder and the person in the corresponding first dangerous detection result, the category of the detection box, and the preset prompt words, to obtain the fourth dangerous detection result corresponding to each to - be - detected picture.

[0026] Through the above - mentioned application embodiments, by using the picture - understanding ability of the multi - modal large model and combining the target positions in the first dangerous detection result, the multi - modal large model is guided to make a correct judgment, making the obtained fourth dangerous detection result more accurate, thus more accurately identifying whether there is dangerous information in the to - be - detected picture, and further more accurately identifying potential safety hazards in dangerous operations, so as to eliminate such potential safety hazards.

[0027] In a second aspect, the present application also provides a target detection device, and the device includes:

[0028] A detection module, configured to use the target detection model to detect N to - be - detected pictures respectively, and obtain the first dangerous detection result corresponding to each to - be - detected picture; where N is a positive integer;

[0029] A screening module, configured to extract K to - be - detected pictures with dangerous information from the N to - be - detected pictures based on the first dangerous detection result; where K is a positive integer less than or equal to N;

[0030] A processing module, configured to use the multi - modal large model to detect the K to - be - detected pictures respectively, and obtain the second dangerous detection result corresponding to each to - be - detected picture.

[0031] In a possible implementation, the target detection model is a small target detection model, and the screening module is further configured to extract M to-be-detected pictures with the first risk detection result indicating the existence of the risk information from the N to-be-detected pictures; where M is a positive integer less than or equal to N; use a multi-target detection model to detect each of the M to-be-detected pictures respectively to obtain a third risk detection result corresponding to each to-be-detected picture; extract K to-be-detected pictures with the third risk detection result indicating the existence of the risk information from the M to-be-detected pictures; where K is a positive integer less than or equal to M.

[0032] In a possible implementation, the processing module is further configured to respectively determine, based on the multi-modal large model including preset rules, whether each of the K to-be-detected pictures conforms to the preset rules; if so, generate a second risk detection result corresponding to the to-be-detected picture including the existence of the risk information; if not, generate a second risk detection result corresponding to the to-be-detected picture including the non-existence of the risk information.

[0033] In a possible implementation, the preset rule is that there is any one of an orange retractable fence, a red retractable fence, a fence of any color, or a construction fence in the to-be-detected picture; or, the preset rule is that there is construction machinery in the corresponding to-be-detected picture; or, the preset rule is that there is a person wearing a safety helmet in the corresponding to-be-detected picture.

[0034] In a possible implementation, the target detection model is a small target detection model or a multi-target detection model, and the processing module is further configured to use the multi-modal large model to process the to-be-detected picture, the detection frame indicating the positional relationship between the ladder and the person in the corresponding first risk detection result, the category of the detection frame, and a preset prompt word to obtain a fourth risk detection result corresponding to each to-be-detected picture.

[0035] In a third aspect, the present application provides an electronic device, including:

[0036] A memory for storing a computer program;

[0037] A processor, configured to implement the steps of the above-mentioned target detection method when executing the computer program stored on the memory.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned target detection method are implemented.

[0039] For the technical effects that can be achieved by each of the above-mentioned second to fourth aspects and each aspect, please refer to the technical effects that can be achieved by the above-mentioned first aspect or various possible solutions in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 FIG. is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application;

[0041] Figure 2 FIG. is a schematic diagram of an application environment provided by an embodiment of the present application;

[0042] Figure 3 FIG. is a schematic flowchart of a target detection method provided by an embodiment of the present application;

[0043] Figure 4 FIG. is a schematic diagram of a processing process of a target detection method provided by an embodiment of the present application;

[0044] Figure 5 FIG. is a schematic diagram of a target detection device provided by an embodiment of the present application;

[0045] Figure 6 FIG. is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The specific operation methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of the present application, "a plurality of" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The connection between A and B can represent: the direct connection between A and B and the connection between A and B through C. In addition, in the description of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0047] Currently, in the detection of whether there is dangerous information in a picture, a single small model (such as the YOLO model) is usually used to detect the picture, and then whether there is dangerous information in the picture is determined according to the dangerous detection result of the picture, resulting in low detection accuracy of the dangerous information in the picture.

[0048] To solve the above problems, this application proposes an object detection method. The object detection model is used to detect N pictures to be detected respectively, and the corresponding first hazard detection results are obtained. Then, based on the first hazard detection results, K pictures to be detected with hazard information are extracted from the N pictures to be detected. Next, the multimodal large model is used to detect the K pictures to be detected respectively, and the corresponding second hazard detection results for each picture to be detected are obtained. Thus, on the basis of the detection of the pictures to be detected by the small object detection model, the N pictures to be detected are preliminarily screened, and then the picture understanding ability of the multimodal large model is used to further detect the K pictures to be detected that are screened out, effectively improving the detection accuracy of the pictures to be detected. Furthermore, it can more accurately identify whether there is hazard information in the pictures to be detected, significantly reducing the false alarm rate of the hazard information in the pictures to be detected, so as to more accurately identify whether there is hazard information in the scene corresponding to the pictures to be detected.

[0049] Figure 1 Shows a schematic diagram of an artificial intelligence main framework, which describes the overall working process of an artificial intelligence system and is applicable to the general requirements of the artificial intelligence field.

[0050] The above artificial intelligence main framework is elaborated below from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis).

[0051] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general processes of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the refinement process of "data - information - knowledge - wisdom".

[0052] The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

[0053] (1) Infrastructure:

[0054] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the external world, and is supported by the basic platform. It communicates with the external world through sensors; the computing power is provided by intelligent chips (hardware acceleration chips such as Central Processing Unit (CPU), Neural-network Processing Unit (NPU), Graphics Processing Unit (GPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), etc.); the basic platform includes relevant platform guarantees and supports such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for computing.

[0055] (2) Data

[0056] The data at the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voices, texts, and also involves the Internet of Things data of traditional devices, including the business data of existing systems and the sensed data such as force, displacement, liquid level, temperature, humidity, etc.

[0057] (3) Data Processing

[0058] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0059] Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on the data.

[0060] Reasoning refers to the process of simulating the intelligent reasoning method of humans in a computer or intelligent system, and using formal information to perform machine thinking and solve problems according to the reasoning control strategy. The typical function is search and matching.

[0061] Decision-making refers to the process of making decisions after the intelligent information is reasoned, and usually provides functions such as classification, sorting, prediction, etc.

[0062] (4) General Capabilities

[0063] After the data is processed through the above-mentioned data processing, some general capabilities can be further formed based on the results of the data processing. For example, it can be an algorithm or a general system. For example, translation, text analysis, computer vision processing, speech recognition, image recognition, and so on.

[0064] (5) Intelligent Products and Industry Applications

[0065] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes actual application. Its application fields mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, intelligent terminal, etc.

[0066] See Appendix Figure 2 In the embodiment of the present invention, a system architecture 200 is provided. The data acquisition device 260 is used to acquire picture data and store it in the database 230. The training device 220 generates a target model / rule 201 based on the picture data maintained in the database 230. Below, how the training device 220 obtains the target model / rule 201 based on the picture data will be described in more detail. The target model / rule 201 can combine the target detection model with the multi-modal large model to process the picture to be detected, and realize the accurate detection of whether there is dangerous information in the picture to be detected.

[0067] The above picture data is the picture to be detected collected, and it can be pictures taken by multiple cameras at different angles or positions at different times (or the same time) of different positions in the scene. For example, based on the pictures taken by 100 high-altitude cameras, one frame is extracted every 10 minutes, then 24 * 6 * 100 = 14400 pictures to be detected can be generated every day. Thus, based on the pictures taken by cameras at different times and different angles, it is detected whether there is dangerous information in the picture to be detected, so as to dynamically detect whether there is dangerous information in the scene.

[0068] The work of each layer in the deep neural network can be described by the mathematical expression as follows: From the physical level, the work of each layer in the deep neural network can be understood as completing the transformation from the input space (the set of input vectors) to the output space (that is, from the row space to the column space of the matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / dimensionality reduction; 2. Magnification / minification; 3. Rotation; 4. Translation; 5. "Bending". Among them, the operations of 1, 2, and 3 are performed by Completion: The operation of 4 is completed by +b, and the operation of 5 is implemented by a(). The reason for using the word "space" here is that the objects to be classified are not individual things, but a class of things. Space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in this vector represents the weight value of a neuron in this layer of the neural network. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to finally obtain the weight matrices of all layers of the trained neural network (the weight matrix formed by many layers of vector W). Therefore, the training process of the neural network is essentially a process of learning the way to control space transformation, and more specifically, learning the weight matrix.

[0069] Because it is hoped that the output of the deep neural network is as close as possible to the value that is really wanted to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network and the really wanted target value, and then according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the really wanted target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (English: Loss Function) or the objective function (English: Objective Function), and they are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0070] The target model / rule obtained by the training device 220 can be applied to different systems or devices. In the appendix Figure 2 In it, the execution device 210 is configured with an I / O interface 212 to interact with external devices, and the "user" can input data to the I / O interface 212 through the client device 240.

[0071] The execution device 210 can call data, code, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.

[0072] The computing module 211 processes the input data using the target model / rule 201. Specifically, first, using the target detection model, it detects N input images to be detected respectively, obtaining the corresponding first hazard detection results; then, based on the first hazard detection results, it extracts K images to be detected with hazard information from the N images to be detected; and then, using the multi-modal large model, it detects the K images to be detected respectively, obtaining the second hazard detection result corresponding to each image to be detected, where K is a positive integer less than or equal to N.

[0073] Finally, the I / O interface 212 returns the processing result to the client device 240 for the user.

[0074] More deeply, the training device 220 can generate corresponding target models / rules 201 based on different data for different targets to provide better results for the user.

[0075] In the case shown in the appendix Figure 2 The user can manually specify the data input into the execution device 210. For example, operate in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data to the I / O interface 212 and obtain the result. If the client device 240 needs user authorization to automatically input data, the user can set the corresponding permissions in the client device 240. The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form can be display, sound, action and other specific ways. The client device 240 can also be used as a data collection end to store the collected image data in the database 230.

[0076] It should be noted that the appendix Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the appendix Figure 2 the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0077] In the embodiments of the present application, the deep neural networks adopted may include a target detection model and a multi-modal large model. The target detection model can be a small target detection model or a multi-target detection model.

[0078] Among them, the detection speed of the small target detection model is faster than that of the multi-target detection model, and the detection speed of the multi-target detection model is faster than that of the multi-modal large model; the detection accuracy of the multi-modal large model is higher than that of the multi-target detection model, and the detection accuracy of the multi-target detection model is higher than that of the small target detection model.

[0079] The model scale of the above-mentioned multi-modal large model (such as model size, number of model parameters) is larger than that of the multi-object detection model, and the model scale of the multi-object detection model is larger than that of the small object detection model. Therefore, the multi-object detection model can also be called the medium model in object detection, and the multi-modal large model can also be called the large model in object detection.

[0080] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0081] Refer to Figure 3 The flowchart of a method for object detection provided by an embodiment of the present application is shown. The method includes:

[0082] S301: Use the object detection model to detect N pictures to be detected respectively, and obtain the first danger detection result corresponding to each picture to be detected.

[0083] Before using the object detection model to detect N pictures to be detected respectively, it is necessary to obtain N pictures to be detected first. Wherein, N is a positive integer.

[0084] Then, use the object detection model to detect N pictures to be detected respectively, and obtain the first danger detection result corresponding to each picture to be detected, so as to complete the preliminary screening of N pictures to be detected.

[0085] Specifically, first use the training device 220 to train the original object detection model to obtain a trained object detection model, which is the current optimal object detection model, so as to improve the detection accuracy of the pictures to be detected. Then, input N pictures to be detected into the trained object detection model respectively. Through the detection of the input data (i.e., a picture to be detected) by the object detection model, the corresponding first danger detection result is obtained. Here, the first danger detection result indicates whether there is danger information in the picture to be detected.

[0086] The above first danger detection result can be that there is danger information or there is no danger information, and can also be information including indicating the existence of danger information or information indicating the non-existence of danger information. In addition, the first danger detection result can also include information such as the position of the detection frame (the position can include coordinates, dimensions), the category of the detection frame, and the confidence level.

[0087] In the embodiments of the present application, the model parameters of the object detection model input by different pictures to be detected can be the same or different.

[0088] In the embodiments of the present application, the above-mentioned target detection model can be a small target detection model, and this small target detection model can be a YOLO model. The threshold (such as confidence) during the detection process of the YOLO model can be flexibly set. For example, a lower confidence is set so that the YOLO model has a low false negative rate for the pictures to be detected with dangerous information, ensuring higher detection accuracy for the pictures to be detected without dangerous information. As a result, when subsequently screening N pictures to be detected based on the first dangerous detection result of the YOLO model, the accuracy of removing the pictures to be detected without dangerous information is higher, further reducing the false positive rate of the pictures to be detected with dangerous information.

[0089] In addition, the above-mentioned target detection model can also be a multi-target detection model, and this multi-target detection model can be a Grounding-dino model, which is convenient for improving the accuracy of the first dangerous detection result of the preliminary screening.

[0090] S302. Based on the first dangerous detection result, extract K pictures to be detected with dangerous information from N pictures to be detected.

[0091] Among them, K is a positive integer less than or equal to N.

[0092] Specifically, among the N first dangerous detection results, screen out the first dangerous detection results with dangerous information. If there are K first dangerous detection results with dangerous information among the N first dangerous detection results, then extract the K pictures to be detected corresponding to these K first detections from the N pictures to be detected. Thus, the preliminary screening of the pictures to be detected with dangerous information among the N pictures to be detected is realized.

[0093] In an alternative embodiment, when the target detection model is a small target detection model, in addition to screening out K pictures to be detected from N pictures to be detected through the above method, K pictures to be detected can also be screened out through the following method:

[0094] Specifically, first, among the N first dangerous detection results, screen out the first dangerous detection results with dangerous information. If there are M (M is a positive integer less than or equal to N) first dangerous detection results with dangerous information among the N first dangerous detection results, then extract the M pictures to be detected corresponding to these M first dangerous detection results from the N pictures to be detected. Thus, the preliminary screening of the pictures to be detected with dangerous information among the N pictures to be detected is realized.

[0095] Then, use the multi-object detection model to detect each of the M images to be detected that have been screened, and obtain the corresponding third hazard detection result for each image to be detected in the M images to be detected. The third hazard detection result can be that there is hazard information or there is no hazard information, and can also be information including information indicating the existence of hazard information or information including information indicating the non-existence of hazard information. In addition, the third hazard detection result can also include information such as the position of the detection frame (the position can include coordinates and dimensions), the category of the detection frame, and the confidence level.

[0096] Taking one image to be detected among the M images to be detected and the multi-object detection model Grounding-dino as an example, input the image to be detected and the target category into Grounding-dino. Through the processing of the input data by Grounding-dino, obtain the corresponding third hazard detection result for this image to be detected.

[0097] Taking the information of three-party construction as the hazard information as an example, the target categories input into Grounding-dino can be texts such as excavators, construction fences, enclosures, and personnel, but are not limited to this.

[0098] Next, from the M third hazard detection results, screen out the third hazard detection results with hazard information. If there are K (K is a positive integer less than or equal to M) third hazard detection results with hazard information among the M third hazard detection results, then from the M images to be detected, extract the K images to be detected corresponding to these K third hazard detection results. Thus, the further screening of the images to be detected with hazard information among the N images to be detected is realized, so as to further improve the detection accuracy and detection speed of the N images to be detected.

[0099] S303. Use the multi-modal large model to detect each of the K images to be detected, and obtain the corresponding second hazard detection result for each image to be detected.

[0100] Among them, the second hazard detection result can be that there is hazard information or there is no hazard information (such as whether there is three-party construction), and can also be information including information indicating the existence of hazard information or information including information indicating the non-existence of hazard information.

[0101] Specifically, for each of the K images to be detected, the following judgment operation is performed:

[0102] Taking one image to be detected among the K images to be detected as an example, based on the multi-modal large model including preset rules, judge whether this image to be detected conforms to the preset rules.

[0103] If it is determined that the to-be-detected picture meets the preset rules, it is determined that there is dangerous information in the to-be-detected picture, that is, a second dangerous detection result corresponding to the to-be-detected picture is generated, and the second dangerous detection result is that there is dangerous information.

[0104] If it is determined that the to-be-detected picture does not meet the preset rules, it is determined that there is no dangerous information in the to-be-detected picture, that is, a second dangerous detection result corresponding to the to-be-detected picture is generated, and the second dangerous detection result is that there is no dangerous information.

[0105] In the embodiments of the present application, the above preset rules may be characteristic objects indicating dangerous information. For example, when the dangerous information is information on three-party construction, the above preset rules may be that there is any one of orange retractable fences, red retractable fences, enclosures of any color, and construction enclosures in the to-be-detected picture; it may also be that there is construction machinery in the to-be-detected picture; it may also be that there are personnel wearing safety helmets in the to-be-detected picture, but it is not limited to this. The specific preset rules can be adjusted according to the specific application scenario, so as to accurately determine the to-be-detected pictures that meet the characteristics of dangerous information (such as the characteristics of three-party construction) according to the settings of the preset rules, so as to accurately determine the to-be-detected pictures with dangerous information (such as information on three-party construction).

[0106] Exemplarily, based on a multi-modal large model including preset rules, a to-be-detected picture is processed to independently judge the following rules according to the picture content. If it meets the requirements, input true, otherwise output false:

[0107] Rule 1: Is there any one of orange retractable fences, red retractable fences, enclosures of any color, and construction enclosures?

[0108] Rule 2: Is there any construction machinery such as excavators, bulldozers, pipe jacking machines, and pile drivers?

[0109] Rule 3: Are there personnel wearing safety helmets?

[0110] Return in JSON format:

[0111] {

[0112] "Rule 1": true / false,

[0113] "Rule 2": true / false,

[0114] "Rule 3": true / false,

[0115] "Reason for judgment": "xxxx"

[0116] }

[0117] Since the preset rules are set based on the feature objects indicating dangerous information, based on the multimodal large model including the preset rules, by judging whether there are feature objects of dangerous information in the picture to be detected, that is, by judging whether the picture to be detected conforms to the preset rules, it is accurately determined whether there is dangerous information in the picture to be detected, so that the pictures to be detected with dangerous information can be more accurately determined among N pictures to be detected.

[0118] In addition, the above preset rules can also be the first preset prompt words. And according to the setting of the first preset prompt words, the second dangerous detection result can also include the category of dangerous information (such as the category of third-party construction) and the reason for the dangerous information (such as the reason for the third-party construction).

[0119] In the embodiment of the present application, the foregoing dangerous information can be information about third-party construction. Thus, by detecting the picture to be detected, it can be accurately detected whether there is information about third-party construction in the picture to be detected, so as to better eliminate the safety hazards of third-party construction.

[0120] In addition, when the application scenario is the identification of safety hazards in dangerous operations, the above dangerous information can also be information about unmanned escalators. Thus, by accurately detecting whether there is information about unmanned escalators in the picture to be detected, the safety hazards of unmanned escalators can be better eliminated. The unmanned escalator means that when working on the ladder, there is no one holding the ladder. For example, when there is a ladder in the picture to be detected and there are people standing on the ladder but no one is holding the ladder, it means that this information is about an unmanned escalator.

[0121] Furthermore, when the dangerous information is information about an unmanned escalator, the object detection method provided by the embodiment of the present application can also be:

[0122] Using the object detection model, each of the N pictures to be detected is detected to obtain the first dangerous detection result corresponding to each picture to be detected. In this case, the object detection model can be a small object detection model, such as the YOLO model; it can also be a multi-object detection model, such as the Grounding-dino model, but it is not limited thereto and can be flexibly adjusted according to the specific application scenario.

[0123] Then, based on the first dangerous detection result, using the multimodal large model, each of the N pictures to be detected is detected to obtain the fourth dangerous detection result corresponding to each picture to be detected. The fourth dangerous detection result can be that there is an unmanned escalator or there is no unmanned escalator, or there is a dedicated escalator or there is no dedicated escalator, or it can include information indicating the existence of an unmanned escalator or information indicating the non-existence of an unmanned escalator, or it can include information indicating the existence of a dedicated escalator or information indicating the non-existence of a dedicated escalator.

[0124] Specifically, the preset prompt (i.e., the second preset prompt), the image to be detected, the detection box indicating the positional relationship between the ladder and the person in the first danger detection result corresponding to the image to be detected, and the category of the detection box are input into the multi-modal large model. Then, using the image understanding ability of the multi-modal large model, the detection box, the category of the detection box, the second preset prompt, and the corresponding image to be detected are comprehensively processed to obtain the corresponding fourth danger detection result.

[0125] In addition, according to the setting of the second preset prompt, the above fourth danger detection result may also include the category of the danger information (such as an unmanned escalator) and the reason for the danger information (such as the reason for the unmanned escalator).

[0126] Through the above method, first, the object detection model is used to detect the image to be detected to obtain the corresponding first danger detection result. Then, using the image understanding ability of the multi-modal large model and combining the target position in the first danger detection result, the multi-modal large model is guided to make a correct judgment, making the obtained fourth danger detection result more accurate, thereby more accurately identifying whether there is an unmanned escalator (or a manned escalator) in the image to be detected, and further more accurately identifying potential safety hazards.

[0127] In summary, for the object detection method proposed in this application, first, the first danger detection result of the object detection model is used to preliminarily screen N images to be detected, removing the images to be detected that definitely do not have danger information. Then, based on the K images to be detected that are initially determined to have danger information, the image understanding ability of the multi-modal large model is used to detect them, thereby reducing the number of image detections of the multi-modal large model. While improving the detection accuracy of the images to be detected, the detection speed is also ensured.

[0128] To further improve the detection accuracy and detection speed of N images to be detected, based on the combination of a small object detection model (such as the YOLO model), a multi-object detection model (such as the Grounding-dino model), and a multi-modal large model, first, the first danger detection result of the small object detection model is used to preliminarily screen N images to be detected, removing the images to be detected that definitely do not have danger information (such as three-party construction). Then, based on the M images to be detected that are initially determined to have danger information (such as three-party construction), the multi-object detection model is used to detect them, thereby further screening the M images to be detected.

[0129] Finally, for the K detected images determined by the multi-object detection model to have dangerous information (such as third-party construction), the multi-modal large model is used to detect them, further reducing the number of image detections by the multi-modal large model. Thus, based on the combination of the small object detection model (such as the YOLO model), the multi-object detection model (such as the Grounding-dino model), and the multi-modal large model, multi-level filtering is performed on the N detected images, further greatly improving the detection accuracy of the detected images with dangerous information, while also ensuring the detection speed, so that potential safety hazards can be eliminated more accurately and efficiently.

[0130] In addition, for the object detection method proposed in this application, when the dangerous information is the information of the escalator without passengers, first use the object detection model (such as the small object detection model or the multi-object detection model) to detect the N detected images, and obtain the first dangerous detection result corresponding to the presence or absence of dangerous information. Then, using the image understanding ability of the multi-modal large model and combining the target positions in the first dangerous detection result of the object detection model, guide the multi-modal large model to make a correct judgment, making the obtained fourth dangerous detection result more accurate, so as to more accurately identify whether there is an escalator without passengers (or whether there is a dedicated escalator) in the detected images, and further more accurately identify potential safety hazards.

[0131] The technical solution of this application will be further described below in combination with the specific application process.

[0132] As Figure 4 shown in the schematic diagram of the processing process of the object detection method, taking the detection of third-party construction as an example, first in the data acquisition module, based on the images captured by 100 high-altitude cameras, one frame is extracted every 10 minutes, and 14,400 images are generated every day. Based on the 14,400 images generated in a day, 14,400 detected images are obtained, that is, N is 14,400. And the 14,400 detected images (i.e., N detected images) are input into the small model detection module to perform preliminary filtering on the 14,400 detected images.

[0133] In the small model detection module, through the YOLO model, 14,400 images to be detected are respectively detected to obtain the first hazard detection result corresponding to each image to be detected. Then, according to the first hazard detection result, the images to be detected with three-party construction are extracted from the 14,400 images to be detected. For example, among the 14,400 first hazard detection results, 8,000 first hazard detection results all include information indicating the existence of three-party construction. Then, from the 14,400 images to be detected, 8,000 images to be detected corresponding to the 8,000 first hazard detection results are extracted, that is, M is 8,000. And the 8,000 images to be detected (i.e., M images to be detected) are input into the medium model detection module to further filter the 8,000 images to be detected.

[0134] In the medium model detection module, through the Grounding-dino model, 8,000 images to be detected are respectively detected to obtain the third hazard detection result corresponding to each image to be detected. Then, according to the third hazard detection result, the images to be detected with three-party construction are extracted from the 8,000 images to be detected. For example, among the 8,000 third hazard detection results, 4,000 third hazard detection results all include information indicating the existence of three-party construction. Then, from the 8,000 images to be detected, 4,000 images to be detected corresponding to the 4,000 third hazard detection results are extracted, that is, K is 4,000. And the 4,000 images to be detected (i.e., K images to be detected) are input into the large model detection module to further detect the 4,000 images to be detected.

[0135] In the large model detection module, using a multi-modal large model including preset rules, 4,000 images to be detected are respectively detected to obtain the second hazard detection result corresponding to each image to be detected, so as to identify whether there is three-party construction in the corresponding images to be detected from the perspective of abstract concepts.

[0136] Through the above method, through the combination of the YOLO model in the small model detection module, the Grounding-dino model in the medium model detection module, and the multi-modal large model in the large model detection module, multi-level filtering is performed on multiple images to be detected, which not only ensures the detection speed of N images to be detected (such as the recognition speed of multiple cameras), but also ensures better detection accuracy (i.e., recognition effect).

[0137] Based on the same inventive concept, an object detection device is also provided in an embodiment of the present application, as Figure 5 shown in the structural schematic diagram of an object detection device provided by the present application. The device includes:

[0138] The detection module 501 is used to detect N pictures to be detected respectively by using a target detection model, and obtain the first dangerous detection result corresponding to each picture to be detected; where N is a positive integer;

[0139] The screening module 502 is used to extract K pictures to be detected with dangerous information from the N pictures to be detected based on the first dangerous detection result; where K is a positive integer less than or equal to N;

[0140] The processing module 503 is used to detect the K pictures to be detected respectively by using a multimodal large model, and obtain the second dangerous detection result corresponding to each picture to be detected.

[0141] In a possible implementation manner, the target detection model is a small target detection model, and the screening module 502 is further used to extract M pictures to be detected with the first dangerous detection result indicating the existence of the dangerous information from the N pictures to be detected; where M is a positive integer less than or equal to N; use a multi-target detection model to detect the M pictures to be detected respectively, and obtain the third dangerous detection result corresponding to each picture to be detected; extract the K pictures to be detected with the third dangerous detection result indicating the existence of the dangerous information from the M pictures to be detected; where K is a positive integer less than or equal to M.

[0142] In a possible implementation manner, the processing module 503 is further used to respectively determine whether each picture to be detected in the K pictures to be detected conforms to the preset rule based on the multimodal large model including the preset rule; if so, generate the second dangerous detection result including the existence of the dangerous information corresponding to the picture to be detected; if not, generate the second dangerous detection result including the non-existence of the dangerous information corresponding to the picture to be detected.

[0143] In a possible implementation manner, the preset rule is that there is any one of an orange retractable fence, a red retractable fence, a fence of any color, or a construction fence in the picture to be detected; or, the preset rule is that there is construction machinery in the corresponding picture to be detected; or, the preset rule is that there is a person wearing a safety helmet in the corresponding picture to be detected.

[0144] In a possible implementation manner, the target detection model is a small target detection model or a multi-target detection model, and the processing module 503 is further used to use the multimodal large model to process the picture to be detected, the detection frame indicating the positional relationship between the ladder and the person in the corresponding first dangerous detection result, the category of the detection frame, and a preset prompt word, and obtain the fourth dangerous detection result corresponding to each picture to be detected.

[0145] Based on the same inventive concept, an electronic device is further provided in an embodiment of the present application. The above-mentioned electronic device can implement the functions of the foregoing target detection device. Refer to Figure 6 , the above-mentioned electronic device includes:

[0146] At least one processor 601 and a memory 602 connected to the at least one processor 601. In the embodiments of the present application, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 Taking the connection between the processor 601 and the memory 602 through the bus 600 as an example. The bus 600 is represented by a thick line in Figure 6 . The connection manners between other components are only for illustrative purposes and are not to be taken as limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 only a thick line is used to represent it in, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and the name is not limited.

[0147] In the embodiments of the present application, the memory 602 stores instructions executable by the at least one processor 601. By executing the instructions stored in the memory 602, the at least one processor 601 can execute the target detection method described above. The processor 601 can implement Figure 5 the functions of each module in the device shown.

[0148] Among them, the processor 601 is the control center of the device. It can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 602 and calling the data stored in the memory 602, various functions of the device and process data, so as to monitor the device as a whole.

[0149] In a possible design, the processor 601 may include one or more processing units. The processor 601 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 can be implemented on the same chip. In some embodiments, they can also be separately implemented on independent chips.

[0150] The processor 601 may be a general-purpose processor, such as a CPU, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the object detection method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0151] The memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 602 may include at least one type of storage medium, for example, it may include flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (abbreviated as RAM in English), a static random access memory (abbreviated as SRAM in English), a programmable read-only memory (abbreviated as PROM in English), a read-only memory (abbreviated as ROM in English), an electrically erasable programmable read-only memory (abbreviated as EEPROM in English), a magnetic memory, a magnetic disk, an optical disk, and so on. The memory 602 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0152] By programming the design of the processor 601, the code corresponding to the object detection method introduced in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 3 the steps of the object detection method of the embodiment shown. How to program the design of the processor 601 is a well-known technology to those skilled in the art and will not be elaborated here.

[0153] Based on the same inventive concept, the embodiments of the present application also provide a storage medium, which stores computer instructions. When the computer instructions run on a computer, the computer is caused to execute the object detection method discussed above.

[0154] In some possible embodiments, various aspects of the object detection method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on a device, the program code is used to cause the control device to execute the steps in the object detection method according to various exemplary embodiments of this application described above in this specification.

[0155] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0156] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0157] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0159] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A target detection method, characterized in that: include: Using the target detection model, N images to be detected are detected respectively to obtain a first danger detection result corresponding to each image to be detected; wherein N is a positive integer; Based on the first danger detection result, extract K pictures to be detected containing danger information from the N pictures to be detected; wherein K is a positive integer less than or equal to N; The K to-be-detected images are detected respectively by using a multimodal large model to obtain a second danger detection result corresponding to each to-be-detected image.

2. The method according to claim 1, characterized in that The target detection model is a small target detection model, and the step of extracting K to-be-detected pictures containing danger information from the N to-be-detected pictures based on the first danger detection result includes: Extracting M pictures to be detected whose first danger detection result is the presence of the danger information from the N pictures to be detected; wherein M is a positive integer less than or equal to N; Using a multi-target detection model, the M images to be detected are detected respectively to obtain a third danger detection result corresponding to each image to be detected; Extract the K pictures to be detected whose third danger detection results are the presence of the danger information from the M pictures to be detected; wherein K is a positive integer less than or equal to M.

3. The method according to any one of claims 1 to 2, characterized in that: The multimodal large model is used to detect the K images to be detected respectively, and a second danger detection result corresponding to each image to be detected is obtained, including: Based on the multimodal large model including preset rules, determining whether each of the K to-be-detected pictures meets the preset rules; If yes, generating the second danger detection result corresponding to the image to be detected and including the danger information; If not, then the second danger detection result corresponding to the image to be detected and including the absence of the danger information is generated.

4. The method according to claim 3, characterized in that The preset rule is that any one of an orange telescopic fence, a red telescopic fence, a fence of any color, and a construction fence exists in the image to be detected; Alternatively, the preset rule is that there is construction machinery in the corresponding image to be detected; Alternatively, the preset rule is that there is a person wearing a helmet in the corresponding image to be detected.

5. The method according to claim 1, characterized in that The target detection model is a small target detection model or a multi-target detection model, and the method further includes: The multimodal large model is used to process the detection frame indicating the positional relationship between the ladder and the person in the image to be detected and the corresponding first danger detection result, the category of the detection frame, and the preset prompt word to obtain a fourth danger detection result corresponding to each image to be detected.

6. A target detection device, characterized in that: include: A detection module, used to detect N to-be-detected pictures respectively using a target detection model, and obtain a first danger detection result corresponding to each to-be-detected picture; wherein N is a positive integer; A screening module, configured to extract K pictures to be detected containing dangerous information from the N pictures to be detected based on the first danger detection result; wherein K is a positive integer less than or equal to N; The processing module is used to use the multimodal large model to detect the K pictures to be detected respectively, and obtain the second danger detection result corresponding to each picture to be detected.

7. The device according to claim 6, characterized in that The target detection model is a small target detection model, and the screening module is further used to extract M pictures to be detected whose first danger detection results are the presence of the danger information from the N pictures to be detected; wherein M is a positive integer less than or equal to N; Using a multi-target detection model, the M images to be detected are detected respectively to obtain a third danger detection result corresponding to each image to be detected; Extract the K pictures to be detected whose third danger detection results are the presence of the danger information from the M pictures to be detected; wherein K is a positive integer less than or equal to M.

8. The device according to any one of claims 6 to 7, characterized in that: The processing module is further used to determine whether each of the K to-be-detected pictures meets the preset rules based on the multimodal large model including the preset rules; If yes, generating the second danger detection result corresponding to the image to be detected and including the danger information; If not, then the second danger detection result corresponding to the image to be detected and including the absence of the danger information is generated.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images