Target detection method, platform, device and equipment based on embedded platform

By deploying processed target recognition algorithms on the embedded platform, the problem of high cost and low efficiency of target detection deployment in security monitoring in the existing technology is solved, and efficient and safe target detection is achieved, which is suitable for scenarios with large traffic. .

CN117372846BActive Publication Date: 2024-11-05HUNAN SUKE INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311344446.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-17
Publication Date
2024-11-05
Estimated Expiration
2043-10-17

AI Technical Summary

Technical Problem

In the security monitoring scenarios, the target detection deployment method based on GPU cloud servers and PCs has problems such as high cost, low efficiency and limited applicability, especially in application scenarios in airports and other venues.

Method used

Using the object detection method based on the embedded platform, the initial object recognition algorithm model is modeled, and the operators that can be supported and cannot be supported are determined, and the replacement is performed. The target recognition algorithm model is deployed on the embedded platform to achieve target information. Real-time detection avoids data transmission delays, and provides lower deployment costs and higher efficiency.

Benefits of technology

It realizes efficient identification of targets on embedded platforms, reduces deployment costs, avoids data transmission delays, improves detection efficiency, and provides a more flexible and secure solution, suitable for security check scenarios with large traffic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117372846B_ABST
    Figure CN117372846B_ABST
Patent Text Reader

Abstract

The present application provides a target detection method, platform, device and equipment based on an embedded platform, wherein the embedded platform deploys a target recognition algorithm model, and the target recognition algorithm model is deployed after the initial target recognition algorithm model is processed according to the network framework of the embedded platform, and the model processing includes: determining the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupported operators to determine the target recognition algorithm model according to the supported operators and the unsupported operators after the replacement processing; the embedded platform obtains the detection image collected by the detection device; based on the target recognition algorithm model and the detection image, the target information in the detection image is recognized, and the target information is fed back to the detection device. Through the above technical solution, compared with the cloud deployment or PC deployment and detection method of the target recognition algorithm in the related technology, its deployment cost is lower and the detection efficiency is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a target detection method, platform, device and equipment based on an embedded platform. Background Art

[0002] The task of object detection is to find all the objects of interest in an image. Object detection is widely used in computer vision applications, such as self-driving cars, drones, security monitoring, etc.

[0003] In security monitoring application scenarios, such as airports, train stations, and subway stations, security inspection equipment collects monitoring images, and security inspectors mainly perform manual image recognition to identify suspicious information (such as suspicious objects or pedestrians). Due to the high labor cost, low efficiency, and poor accuracy, the related technology implements the automatic detection of suspicious information during the security inspection process by deploying the target recognition algorithm in the graphics processing unit (GPU) cloud server or in the personal computer (PC).

[0004] However, the deployment method based on GPU cloud servers requires the construction of private cloud servers, and during the target detection process, data needs to be uploaded to the cloud, resulting in extremely high deployment costs and low detection efficiency. The PC-based target recognition algorithm deployment method requires the adaptation of hardware such as graphics cards and video acquisition, which has a high deployment cost, and the PC is too large to be used in strictly restricted application scenarios such as airports and railways, resulting in limited application scenarios. Summary of the invention

[0005] The present application provides a target detection method, platform, device and equipment based on an embedded platform to solve at least one of the above technical problems.

[0006] According to one aspect of the present application, a target detection method based on an embedded platform is provided, wherein the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing comprises: determining supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and performing replacement processing on the unsupportable operators, so as to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing;

[0007] The method is applied to an embedded platform, which is connected to a detection device. The method includes: acquiring a detection image collected by the detection device; identifying target information in the detection image based on a target recognition algorithm model, wherein the target information includes target object information or target pedestrian information; feeding back the target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the target information, and displays the marked detection image on a display interface.

[0008] In one embodiment, the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model are determined, including: parsing the initial target recognition algorithm model to obtain a static calculation graph, the static calculation graph including all first operators of the initial target recognition algorithm model; traversing all first operators in the static calculation graph, and determining each first operator as a supported operator or an unsupported operator according to a preset operator support table of the network framework.

[0009] In one embodiment, a target recognition algorithm model is determined based on supported operators and unsupported operators after replacement processing, including: using supported operators and unsupported operators after replacement processing as all second operators in a static calculation graph; traversing all second operators in the static calculation graph to determine the computing resources required for each second operator to be deployed in a network framework; replacing second operators whose computing resources reach a first preset threshold to obtain all third operators in the static calculation graph, and determining the target recognition algorithm model based on the third operators.

[0010] In one embodiment, determining a target recognition algorithm model based on a third operator includes: identifying a set of operators in the constant layer of the third operator, deleting the set of operators, obtaining all fourth operators in the static calculation graph, and determining the target recognition algorithm model based on the fourth operator; or, fusing the third operator with similar operators according to the category of the third operator, obtaining all fifth operators in the static calculation graph, and determining the target recognition algorithm model based on the fifth operator.

[0011] In one embodiment, after acquiring the detection image collected by the detection device and before performing target recognition on the detection image based on the target recognition algorithm model, it also includes:

[0012] Performing image preprocessing on the detection image, the image preprocessing includes at least one of the following: image proportional scaling, color filling and color conversion; converting the detection image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supported operators and the unsupported operators after replacement processing in the target recognition algorithm model; identifying the target information in the detection image based on the target recognition algorithm model, including: inputting the target input data into the target recognition algorithm model to identify the target information in the detection image.

[0013] In one embodiment, the target information carries target detection frame information; after obtaining the target information, it also includes: parsing the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame; based on the detection frame category and the detection frame score, filtering out the target detection frames in the corresponding detection frame category of each target detection frame whose detection frame scores are lower than a second preset threshold; based on the detection frame coordinates, determining the coordinates of the filtered target detection frames on the detection image.

[0014] According to a second aspect of the present application, a target detection method based on an embedded platform is provided, wherein the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing comprises: determining supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and performing replacement processing on the unsupportable operators, so as to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing;

[0015] The method is applied to a detection device, which is connected to an embedded platform. The method includes: acquiring a detection image and sending the detection image to the embedded platform, so that the embedded platform recognizes target information in the detection image based on a target recognition algorithm model, and feeds the target information back to the detection device; marking corresponding target objects or target pedestrians in the detection image according to the target information, and displaying the marked detection image on a display interface.

[0016] According to a third aspect of the present application, an embedded platform and an embedded platform target recognition algorithm model are provided, wherein the target recognition algorithm model is deployed after model processing of an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing comprises: determining supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing; the embedded platform is connected to a detection device, and the embedded platform comprises:

[0017] An acquisition unit is configured to acquire a detection image collected by a detection device; an identification unit is configured to identify target information in the detection image based on a target recognition algorithm model, wherein the target information includes target object information or target pedestrian information; a transmission unit is configured to feed back the target information to the detection device so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the target information, and displays the marked detection image on a display interface.

[0018] According to a fourth aspect of the present application, a detection device is provided, the detection device is connected to an embedded platform, and the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing includes: determining supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing;

[0019] The detection device is connected to the embedded platform, and the detection device includes: an acquisition unit, which is configured to acquire a detection image and send the detection image to the embedded platform, so that the embedded platform recognizes the target information in the detection image based on the target recognition algorithm model, and feeds the target information back to the detection device; a marking and display unit, which is configured to mark the corresponding target object or target pedestrian in the detection image according to the target information, and display the marked detection image on the display interface.

[0020] According to a fifth aspect of the present application, there is provided an electronic device, comprising: a memory and a processor;

[0021] Memory stores computer-executable instructions;

[0022] The processor executes the computer execution instructions stored in the memory, so that the electronic device executes the target detection method based on the embedded platform.

[0023] The target detection method, platform, device and equipment based on the embedded platform provided by the present application can effectively solve the difficulty of directly deploying the trained model, or the problem that the embedded platform is difficult to execute the currently trained model algorithm after deployment, by deploying the target recognition algorithm after model processing in the embedded platform, thereby achieving the purpose of efficient recognition of the embedded platform; compared with the related technology, the embedded platform is used to deploy the target recognition algorithm model for target detection, and there is no need to build a private cloud server and adapt the graphics card. The embedded device can complete the deployment of the algorithm, and its deployment cost is lower than that of the cloud server and the PC; and by deploying the target recognition algorithm model on the embedded device, the target recognition can be realized. Real-time detection of target information avoids the time loss caused by data transmission and processing delays, accelerates algorithm deployment and application, and makes the detection process more efficient. In addition, embedded devices can directly detect targets on site without uploading sensitive data to cloud servers for processing, which can effectively protect personal privacy and security-sensitive information, making the detection process safer. In addition, the use of embedded devices to deploy target recognition algorithms can be customized according to different detection needs and scenarios, providing more flexible and scalable solutions. Embedded devices usually have lower power consumption and miniaturization, which can save energy and space resources, and can run stably for a long time, making them more suitable for security inspection scenarios with large traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] Figure 1 A possible scenario diagram provided for an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a flow chart of a target detection method based on an embedded platform provided in an embodiment of the present application;

[0027] Figure 3 This is an example diagram of a static calculation graph in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of another target detection method based on an embedded platform provided in an embodiment of the present application;

[0029] Figure 5 A flowchart of another target detection method based on an embedded platform provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of another target detection method based on an embedded platform provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an embedded platform provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a detection device provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Fig.10 It is a structural block diagram of a terminal device provided by an exemplary embodiment of the present application.

[0035] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0037] To facilitate understanding of the embodiments of the present application, the following terms are first explained:

[0038] PyTorch: An open source Python machine learning library, which is a possible algorithm training framework used in the embodiments of the present application and is used to train the target recognition algorithm model.

[0039] TensorRT: is an open source, high-performance deep learning inference optimizer that can provide low-latency, high-throughput deployment inference for deep learning applications. It is a possible algorithm deployment framework adopted by the embedded platform of the embodiment of this application. Deploying the trained target recognition algorithm model to TensorRT can achieve algorithm inference acceleration.

[0040] Static calculation graph: The structure of the calculation graph is determined before the program runs, which means that the structure of the calculation graph will not change during runtime. In other words, the static calculation graph is to fix all calculation parameters before the algorithm is calculated, and all parameters will not change during the calculation process. Compared with the dynamic calculation graph, the parameters can be changed during the calculation process.

[0041] CUDA: A computing platform, which is an open source general-purpose parallel computing architecture that can solve complex computing problems. It includes the CUDA instruction set architecture (ISA) and the parallel computing engine inside the GPU. Developers can use C language to write programs for the CUDA architecture, and the programs written can run with ultra-high performance on processors that support CUDA. It is a possible pre-processing and post-processing platform used in the embodiments of this application.

[0042] The following is an analysis of the existing technical problems and the technical concepts of the embodiments of the present application:

[0043] Take the application of target detection technology in the field of security inspection and control as an example. Traditional security inspection and control methods mainly rely on manual image judgment. The detection accuracy is limited by the subjective consciousness of the security inspectors, and the speed is limited by the workload of the security inspectors, resulting in extremely low efficiency of manual detection. By applying target detection technology to the field of security inspection and control, the detection accuracy and efficiency of suspicious information (such as suspicious objects or pedestrians) can be effectively improved based on the target recognition algorithm, while significantly reducing labor costs.

[0044] Since traditional security inspection machines do not have intelligent recognition capabilities and the security inspection machines have complete systems that are not conducive to secondary development, related technologies use target recognition algorithms in the graphics processing unit (GPU) cloud server or in the personal computer (PC) to achieve automatic detection of suspicious information during the security inspection process.

[0045] In a deployment solution based on a GPU cloud server, for example, a screening port connected to the interior is provided in the security inspection device, a monitoring machine is installed on the security inspection device to monitor the luggage situation in the security inspection device, a screening mechanism is provided in the security inspection device to transport luggage with problems out of the screening port, and a cloud server is respectively connected to the monitoring machine and the screening mechanism by signal. The cloud server is used to analyze and process the luggage image monitored by the monitoring machine, and transmit the processed image data to the monitoring machine and the screening mechanism, so as to drive the screening mechanism to transport the luggage with problems out of the screening port. Obviously, this deployment plan is prone to the following problems: cost issues. The cost of setting up a private cloud server is extremely high; delay issues. Since the contraband detection algorithm needs to upload images to the cloud platform for processing and then return the results, the network bandwidth limitation will cause delays in the detection process, slowing down the security check; security inspection machine network connection issues. After the algorithm is deployed to the cloud platform, the security inspection equipment needs to always stay connected to the cloud. If there is a problem or interruption in the network, contraband detection will be impossible, reducing the reliability of security inspections; data security issues: uploading contraband images to the cloud platform for processing will involve user privacy and sensitive data. If the cloud platform's data security measures are insufficient, there will be a risk of data leakage.

[0046] In the deployment scheme based on PC, for example, according to the usage scenario, the working parameters of the X-ray security inspection machine are set through the virtual instrument platform built on PC; the X-ray security inspection machine is used to collect the X-ray image video stream and the collected X-ray image video stream is input into the virtual instrument platform built on PC; the X-ray image video stream is processed by the virtual instrument platform, and the X-ray image is intercepted from the X-ray image video stream and input into the trained convolutional neural network; the convolutional neural network is used to detect and identify contraband in the X-ray image in the interception order; when contraband is detected, the contraband in the X-ray image is marked and displayed on the virtual instrument platform. Obviously, this deployment scheme is prone to the following problems: cost problem, the deployment cost based on PC is high, and hardware such as graphics card and video acquisition module are required; adaptability problem, the PC is too large, and it is not suitable for application scenarios with strict restrictions in airports, railways and other places, and it is not easy to integrate with other security inspection equipment.

[0047] In view of this, the embodiments of the present application provide a target detection method, platform, device and equipment based on an embedded platform. In a possible application scenario, the technical solution can be applied to the application scenario of security monitoring. By way of example, the executor of the method provided in the embodiments of the present application can be an embedded platform or a detection device. More specifically, for example, it can be an embedded platform or a detection device set up in a security inspection and control site. Optionally, the embedded platform can adopt the TensorRT network framework, and the detection device can adopt a security inspection machine; wherein, the embedded platform can be embedded in any device in the security inspection and control site, for example, embedded in the detection device, or embedded in other devices other than the detection device, such as monitoring devices and other equipment. The embodiments of the present application do not specifically limit the embedding position of the embedded device.

[0048] The embodiment of the present application deploys a target recognition algorithm model by utilizing an embedded platform, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing includes: determining the supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupported operators after the replacement processing; the method is applied to the embedded platform, and the embedded platform is connected to a detection device, and the method includes: acquiring a detection image collected by the detection device; identifying target information in the detection image based on the target recognition algorithm model, the target information including target object information or target pedestrian information; feeding back the target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the target information, and displays the marked detection image on a display interface.

[0049] In this process, according to the characteristics of the network framework adopted by the embedded platform, by processing the model of the initial target recognition algorithm and deploying it in the embedded platform, it can effectively solve the problem that the trained model cannot be directly deployed to the embedded platform, or the embedded platform is difficult to execute the currently trained model algorithm after deployment; using the embedded platform to deploy the target recognition algorithm model for target detection, there is no need to build a private cloud server and adapt the graphics card, the embedded device can complete the deployment of the algorithm, and its deployment cost is lower than that of cloud servers and PCs; and by deploying the target recognition algorithm model on the embedded device, real-time detection of target information is achieved, avoiding It eliminates the time loss caused by data transmission and processing delays, accelerates algorithm deployment and application, and makes the detection process more efficient. In addition, embedded devices can directly perform target detection on site without uploading sensitive data to cloud servers for processing, which can effectively protect personal privacy and security-sensitive information, making the detection process safer. In addition, the use of embedded devices to deploy contraband detection algorithms can be customized according to different detection needs and scenarios, providing more flexible and scalable solutions. Embedded devices usually have lower power consumption and miniaturization, which can save energy and space resources, and can run stably for a long time, making them more suitable for security inspection scenarios with large traffic.

[0050] The following uses the application in security monitoring scenarios and identification of prohibited items as an example, and combines the accompanying drawings and specific embodiments to explain in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. It should be noted that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0052] An embodiment of the present application provides a target detection method based on an embedded platform, wherein the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing includes: determining supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupported operators to determine the target recognition algorithm model according to the supported operators and the unsupported operators after the replacement processing.

[0053] In this embodiment, the initial target recognition algorithm is a trained model. Optionally, the PyTorch algorithm training framework is used to train the initial target recognition algorithm model for identifying contraband. The training process of the initial target recognition algorithm can refer to the prior art and will not be described in detail here.

[0054] In this embodiment, the embedded platform can use NVIDIA Jetson Nano, and the network framework of the embedded platform can use TensorRT, where Jetson Nano has a low cost, the TensorRT reasoning framework is relatively mature, and Jetson nano supports the TensorRT network framework, and its computing power can meet the reasoning of target recognition algorithms such as YoloV8, which can achieve more efficient target recognition. In some embodiments, other embedded platforms or other network frameworks such as NCNN can also be used. This embodiment does not specifically limit the embedded platform and the network framework it uses.

[0055] In this embodiment, the operator (OP for short) is the computing unit in the model. Those skilled in the art can understand that the deep learning algorithm is composed of computing units, which are also operators. In the network model, the operator corresponds to the computing logic in the layer, for example: the convolution layer is an operator; the weight summation process in the fully connected layer (FC layer) is an operator. Another example: tanh, ReLU, etc. are operators used as activation functions in the network model. For different deployment frameworks, the operators they support are also different. Especially for embedded platforms, based on their small size and limited computing resources, if the trained model is directly deployed to the embedded platform, it is easy to cause the embedded platform to be difficult to execute the currently trained model algorithm. In order to avoid the above problems, this embodiment performs model processing on the initial target recognition algorithm and then deploys it in the embedded platform, so that the embedded platform can execute an efficient target recognition process.

[0056] In one implementation, the application scenario may include multiple detection devices at the same time, optionally, such as Figure 1 As shown, it includes multiple security inspection machines 11. The embedded platform 12 deployed with the target recognition algorithm model can be embedded in a security inspection machine 11 and connected to other security inspection machines 11 via a wired or wireless network to receive detection images collected by other security inspection machines 11 and perform target recognition. In other implementations, in order to balance recognition efficiency and operating costs, the number of embedded devices and security inspection machines can be allocated according to a certain ratio, such as Figure 1In the example, every three security inspection machines are connected to an embedded platform, which can be used to process the detection images of the three security inspection machines and perform target recognition.

[0057] Please refer to Figure 2 , Figure 2 This is a flow chart of a target detection method based on an embedded platform provided in the present application. The method is applied to an embedded platform 11. The embedded platform 11 is connected to a detection device 12. The method includes steps S201-S203.

[0058] Step S201: Acquire a detection image collected by a detection device.

[0059] In one possible implementation, the detection device carries a camera function, or a camera is integrated in the detection device, and the detection image is collected by using the camera. After collecting the detection image, the detection device transmits the corresponding detection image to the embedded platform in real time, and the embedded platform obtains the detection image transmitted by the detection device, wherein the transmission mode can be internal transmission or external transmission.

[0060] Step S202: identifying target information in the detection image based on a target recognition algorithm model, where the target information includes target object information or target pedestrian information.

[0061] It should be noted that the target items may be prohibited items specified in the security monitoring site, such as sprays and the like, and the target pedestrians may be pedestrians whose passage is restricted in the security monitoring site.

[0062] Specifically, after the embedded platform deploys the target recognition algorithm, the target recognition algorithm model uses its computing resources to perform the calculations of each operator to obtain the corresponding target detection results. It can be understood that, taking contraband recognition as an example, the target recognition model is trained based on the corresponding contraband. After the model deployment is completed, the target recognition algorithm model can quickly train the corresponding target information in the detection image, that is, the contraband information, and the same applies to other target item information and target pedestrian information.

[0063] Step S203: Feedback the target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the target information, and displays the marked detection image on the display interface.

[0064] In this embodiment, after the embedded platform completes the target information recognition of the detection image, it feeds back the recognized target information to the detection device, and the detection device further marks the target object or target pedestrian in the detection image to facilitate the security monitoring management personnel to make further confirmation and processing actions.

[0065] It should be noted that the display interface in this embodiment can be the display interface of the detection device itself, or it can be a display interface transmitted to other devices via wired or wireless transmission, such as the security monitoring personnel’s mobile phone or computer or other devices with display function.

[0066] This embodiment uses an embedded platform for target detection. There is no need to build a private cloud server and adapt the graphics card. The embedded device can complete the deployment of the algorithm. Compared with cloud servers and PCs, its deployment cost is lower. Moreover, by deploying the illegal target recognition algorithm model on the embedded device, real-time detection of target information is achieved, avoiding the time loss caused by data transmission and processing delays, accelerating algorithm deployment and application, and its detection process is more efficient. Moreover, the embedded device can directly perform target detection on site, without uploading sensitive data to the cloud server for processing, which can effectively protect personal privacy and security sensitive information, and its detection process is safer. In addition, the use of embedded devices to deploy contraband detection algorithms can be customized according to different detection requirements and scenarios, providing a more flexible and scalable solution, and embedded devices usually have the characteristics of low power consumption and miniaturization, which can save energy and space resources, and can run stably for a long time, and are more suitable for security inspection scenarios with large traffic.

[0067] In one implementation, determining the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model may include the following steps:

[0068] Parsing the initial target recognition algorithm model to obtain a static calculation graph, the static calculation graph includes all first operators of the initial target recognition algorithm model;

[0069] All first operators in the static computation graph are traversed, and each first operator is determined as a supported operator or an unsupported operator according to a preset operator support table of the network framework.

[0070] In this embodiment, taking the initial target recognition algorithm model trained with PyTorch as an example, by loading and parsing the PyTorch model, the static calculation graph of the entire model can be obtained. Among them, the operator is a calculation unit, an operator is responsible for a calculation logic, and the model is composed of many operators connected layer by layer to form a complete static calculation graph. The static calculation graph is as follows Figure 3 shown.

[0071] Those skilled in the art can understand that the preset operator support table is known and determined after the network framework is determined. In other words, different network frameworks correspond to different preset operator support tables. Taking the network framework as TensorRT as an example, for different versions of TensorRT (corresponding to different network frameworks), the operator support table is also inconsistent. Taking the operator support table of a certain version of TensorRT (due to the large number of supported operators, this embodiment only illustrates some supported operators) as an example, as shown in the following table:

[0072]

[0073] In the above table, "Y" corresponds to supported operators, and "N" corresponds to unsupported operators. It can be seen that the TensorRT network framework does not support the fully connected layer (FC) operator and the deconvolution unsample operator, while the trained initial target recognition model contains these two or more operators that are not supported by the embedded platform network framework. In order to achieve efficient deployment of the algorithm, this embodiment replaces the operators that are not supported by the network framework with supported operators. For example, the FC operator is replaced by the conv operator supported by TensorRT, and the Unsample operator is replaced by the convtranspose operator supported by TensorRT.

[0074] Optionally, the FC operator is replaced with the conv operator. By setting the corresponding parameters, the same input and output can be obtained. For example, if the input is c * n * n, the convolution with n * n convolution kernel can be used to keep the input and output consistent. Upsample is replaced with ConvTranspose. Upsample inserts new values ​​into the input tensor to obtain a new tensor with an enlarged shape. Interpolation is implemented in a specific way, such as (nearest neighbor interpolation, linear interpolation, etc.). The same effect can be achieved by constructing a convtranspose. For example, if the input is b * c * n *n, the output is b * c * (2*n) * (2*n) after being doubled. The convtranspose with a 4*4 convolution kernel can be used to keep the input and output consistent.

[0075] Optionally, unsupported operators may include operators that are not supported by dynamic shape, where the dynamic shape is an operator whose shape of Tensor depends on specific operations and cannot be calculated in advance. If there are operators that are not supported by TensorRT, the algorithm cannot be deployed to the TensorRT platform or the computing power is not supported, resulting in the inability to achieve the recognition effect. In order to deploy the model on TensorRT or improve the computing efficiency of the platform, replace the unsupported operators. An unsupported operator can be realized by converting one supported operator or combining multiple supported operators. Similarly, operators that are not supported by dynamic shape can also be realized by the above method.

[0076] Furthermore, after completing the operator replacement, operator fusion operations can also be performed, such as conv + bn fusion; conv + bn + activation layer (including: relu, mish, silu, etc.) fusion; conv + bn + add / mul / div fusion; merge the repeated operators before and after (for example, two reshapes are merged into one reshape) to reduce redundant calculations, further achieve the effect of optimization and acceleration, and meet the target recognition needs of security monitoring.

[0077] In other embodiments, the supported operators and unsupported operators determined between the network framework of the embedded platform and the initial target recognition algorithm model are similarly described.

[0078] In one embodiment, determining the target recognition algorithm model according to the supported operators and the unsupported operators after replacement processing may include the following steps:

[0079] The supported operators and the replaced unsupported operators are used as all the second operators in the static computation graph;

[0080] Traverse all second operators in the static computation graph and determine the computing resources required for each second operator to be deployed in the network framework;

[0081] The second operator whose computing resources reach the first preset threshold is replaced to obtain all the third operators in the static computing graph, and the target recognition algorithm model is determined according to the third operators.

[0082] In this embodiment, special operators that occupy a large amount of memory are replaced by static calculation graphs after replacement processing. These special operators, that is, operators that use additional computing sources in TensorRT, occupy a large amount of additional memory in actual use. In order to avoid the appearance of these operators in the final TensorRT model affecting the recognition efficiency of the model, they are replaced by other operators. The replacement method can refer to the above replacement example, which will not be repeated here.

[0083] In one implementation, determining the target recognition algorithm model according to the third operator may include the following steps:

[0084] Identify the set of operators in the constant layer in the third operator, delete the set of operators, obtain all fourth operators in the static calculation graph, and determine the target recognition algorithm model based on the fourth operators; or, fuse the third operators of the same type according to the category of the third operator, obtain all fifth operators in the static calculation graph, and determine the target recognition algorithm model based on the fifth operator.

[0085] In this embodiment, the constant layer can be regarded as a fixed value, and the final result of the calculation process of the constant layer is still the constant layer. In this embodiment, by deleting the operator set of the constant layer, unnecessary calculation processes can be avoided, thereby achieving the purpose of further performance optimization.

[0086] In this embodiment, after completing the above-mentioned operations such as operator replacement and deletion, the operators are fused, wherein the same type of operators, that is, operators that can perform the same type of operations, for example, operators that can all perform linear operations, wherein an example of operator fusion is as follows:

[0087] conv + bn + activation layer (including: relu, mish, silu, etc.) fusion; conv + bn + add / mul / div fusion; merge repeated operators before and after (for example: two reshapes are merged into one reshape). Among them:

[0088] The conv+bn fusion principle is that during inference, the parameters of each operator are fixed, and conv and bn can be regarded as a function: f(x) = w_conv * x + b_conv; conv+bn fusion is regarded as a new function: g(f(x)) = w_bn * (w_conv * x + b_conv) + b_bn = (w_bn * w_conv) * x + w_bn *; b_conv +b_bn; w_bn * w_conv is used as the new weight w, and w_bn * b_conv + b_bn is used as the new bias b;

[0089] The fusion principle of conv+bn+add is to fuse the parameters of add with the bias b of conv+bn to obtain a new bias b_new;

[0090] The fusion principle of conv+bn+mul / div is to fuse the parameters of add with the weight w of conv+bn to obtain a new bias w_new.

[0091] In this embodiment, by calling the TensorRT API, the PyTorch operator is replaced with the TensorRT operator to form a complete static calculation graph, and the dynamic shape is set, and the additional computing source is closed to reduce memory usage. Finally, the TensorRT model is built to achieve the purpose of efficient deployment of the target recognition algorithm on the embedded platform.

[0092] Please refer to Figure 4 , Figure 4 A flowchart of another target detection method based on an embedded platform provided in an embodiment of the present application is provided. Based on the above embodiment, this embodiment can effectively reduce the occupation of model computing resources by preprocessing the image outside the model, thereby improving the target recognition efficiency. Specifically, after obtaining the detection image collected by the detection device (step S201), and before performing target recognition on the detection image based on the target recognition algorithm model (step S202), the following steps S401 and S402 can also be included, and step S202 can be further divided into step S202a.

[0093] Step S401: perform image preprocessing on the detection image, where the image preprocessing includes at least one of the following: image scaling, color filling and color conversion.

[0094] In actual applications, the original inspection images collected by security inspection machines usually have large memory and colors that are not easy to directly identify. If the target recognition algorithm model of the embedded platform is directly used for target recognition, it will take up more computing resources, which will affect the efficiency of target recognition to a certain extent. Image scaling can ensure that the image fed into the algorithm (i.e., the target recognition algorithm model) will not be deformed, while color filling operations can ensure that the image size fed into the algorithm meets the requirements of the algorithm, and color conversion can ensure that the color space of the image meets the requirements of the algorithm.

[0095] In this embodiment, after acquiring the detection image and before using the target recognition algorithm model to perform target recognition, the embedded platform first performs image preprocessing on the detection image to improve the detection efficiency. In one implementation, the image preprocessing, that is, the preprocessing of the target recognition algorithm model, can be implemented based on a graphics card (Compute Unified Device Architecture, CUDA). The algorithm preprocessing implemented based on CUDA can run on a GPU, which is much faster than preprocessing on the target recognition algorithm model, and can also reduce resource usage, thereby achieving the purpose of improving the target recognition efficiency of the embedded platform.

[0096] Step S402, converting the detection image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supported operators and the unsupported operators after replacement processing in the target recognition algorithm model.

[0097] S202a, inputting the target input data into the target recognition algorithm model to identify the target information in the detection image.

[0098] In this embodiment, after the pre-processed image is processed, the corresponding image data is converted into input data required by the algorithm, such as edge information (gradient vector) of the image and other data.

[0099] This embodiment performs pre-processing on the detection image and then completes the loading and reasoning operations of the TensorRT model on the embedded device, which can effectively reduce the processing difficulty of the model and optimize the target recognition efficiency.

[0100] Please refer to Figure 5 , Figure 5 A flowchart of another target detection method based on an embedded platform provided in an embodiment of the present application is provided. Based on the above embodiment, this embodiment performs post-processing on the target information after the model performs target recognition, thereby improving the recognition accuracy rate on the basis of improving the efficiency of model recognition. Specifically, the target information carries target detection frame information; after obtaining the target information in step S202, the following steps S501-S503 may also be included.

[0101] Step S501, parsing the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame;

[0102] Step S502: based on the detection frame category and the detection frame score, filter out target detection frames whose detection frame scores in the corresponding detection frame category are lower than a second preset threshold among the target detection frames;

[0103] Step S503: Based on the detection frame coordinates, determine the coordinates of the filtered target detection frame on the detection image.

[0104] In practical applications, after completing target detection, the target recognition algorithm model may generate detection boxes for multiple positions in the image. If the model is directly used to screen and locate the detection boxes, the processing workload of the model will be large, which will be difficult to meet the miniaturization requirements of embedded devices.

[0105] After the model of this embodiment completes the target recognition, the embedded device can combine other algorithms such as CUDA to post-process the target information, use CUDA to filter a part of the target detection frame, and determine the coordinate position of the filtered target detection frame in the detection image. The coordinate position is carried in the target information and fed back to the detection device. The detection device can quickly mark the corresponding target object or target pedestrian, and display the marked detection image on the display interface to improve the recognition accuracy.

[0106] In this embodiment, the CUDA algorithm is used for post-processing, which improves the processing rate compared to post-processing through the target recognition algorithm model and can reduce the resource usage of the target recognition algorithm model. Because the result of algorithm reasoning cannot be directly used as the real result and contains a lot of unnecessary information, the result of algorithm reasoning needs additional post-processing, which mainly includes filtering out the detection box below the set threshold, restoring the detection box coordinates predicted by the algorithm to the original image coordinates, non-maximum suppression (NMS), etc.

[0107] The embodiment of the present application also provides a target detection method based on an embedded platform, wherein the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after the initial target recognition algorithm model is processed according to the network framework of the embedded platform, and the model processing includes: determining the supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing;

[0108] The method is applied to a detection device, which is connected to an embedded platform, such as Figure 6 As shown, the method may include the following steps:

[0109] Step S601, collecting a detection image, and sending the detection image to the embedded platform, so that the embedded platform recognizes the target information in the detection image based on the target recognition algorithm model, and feeds the target information back to the detection device;

[0110] Step S602: annotate the corresponding target object or target pedestrian in the detection image according to the target information, and display the annotated detection image on the display interface.

[0111] In one implementation, determining the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model includes:

[0112] Parsing the initial target recognition algorithm model to obtain a static calculation graph, the static calculation graph includes all first operators of the initial target recognition algorithm model;

[0113] All first operators in the static computation graph are traversed, and each first operator is determined as a supported operator or an unsupported operator according to a preset operator support table of the network framework.

[0114] In one embodiment, determining a target recognition algorithm model according to the supported operators and the replaced unsupported operators includes:

[0115] The supported operators and the replaced unsupported operators are used as all the second operators in the static computation graph;

[0116] Traverse all second operators in the static computation graph and determine the computing resources required for each second operator to be deployed in the network framework;

[0117] The second operator whose computing resources reach the first preset threshold is replaced to obtain all the third operators in the static computing graph, and the target recognition algorithm model is determined according to the third operators.

[0118] In one implementation, determining a target recognition algorithm model according to the third operator includes:

[0119] Identify the operator set in the constant layer in the third operator, delete the operator set, obtain all the fourth operators in the static calculation graph, and determine the target recognition algorithm model according to the fourth operator;

[0120] or,

[0121] The third operator is fused with similar operators according to the category of the third operator to obtain all fifth operators in the static calculation graph, and the target recognition algorithm model is determined according to the fifth operator.

[0122] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the steps in the method embodiment of the above-mentioned embedded platform end, and no further details will be given here.

[0123] The embodiment of the present application also provides an embedded platform and an embedded platform target recognition algorithm model, wherein the target recognition algorithm model is deployed after the initial target recognition algorithm model is processed according to the network framework of the embedded platform, and the model processing includes: determining the supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing; the embedded platform is connected to the detection device, such as Figure 7 As shown, the embedded platform 70 includes:

[0124] An acquisition unit 71, which is configured to acquire a detection image collected by a detection device;

[0125] An identification unit 72, which is configured to identify target information in the detection image based on a target recognition algorithm model, where the target information includes target object information or target pedestrian information;

[0126] The transmission unit 73 is configured to feed back the target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the target information, and displays the marked detection image on the display interface.

[0127] In one embodiment, the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model are determined specifically as follows: the initial target recognition algorithm model is parsed to obtain a static calculation graph, which includes all first operators of the initial target recognition algorithm model; all first operators in the static calculation graph are traversed, and each first operator is determined as a supported operator or an unsupported operator according to a preset operator support table of the network framework.

[0128] In one embodiment, a target recognition algorithm model is determined based on supported operators and unsupported operators after replacement processing, specifically: supporting operators and unsupported operators after replacement processing are used as all second operators in a static calculation graph; all second operators in the static calculation graph are traversed to determine the computing resources required for each second operator to be deployed in the network framework; second operators whose computing resources reach a first preset threshold are replaced to obtain all third operators in the static calculation graph, and the target recognition algorithm model is determined based on the third operators.

[0129] In one embodiment, the target recognition algorithm model is determined according to the third operator, specifically: identifying the operator set in the constant layer of the third operator, and deleting the operator set to obtain all fourth operators in the static calculation graph, and determining the target recognition algorithm model according to the fourth operator; or, fusing the third operator of the same type according to the category of the third operator to obtain all fifth operators in the static calculation graph, and determining the target recognition algorithm model according to the fifth operator.

[0130] In one embodiment, the device further comprises:

[0131] A preprocessing unit, which is configured to perform image preprocessing on the detection image, wherein the image preprocessing includes at least one of the following: image scaling, color filling and color conversion;

[0132] A conversion unit, which is configured to convert the detection image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supportable operators and the unsupportable operators after replacement processing in the target recognition algorithm model;

[0133] The recognition unit is specifically configured to input the target input data into the target recognition algorithm model to recognize the target information in the detection image.

[0134] In one implementation, the target information carries target detection frame information;

[0135] The device also includes:

[0136] A parsing unit, configured to parse the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame;

[0137] A filtering unit, configured to filter out target detection frames whose detection frame scores in the corresponding detection frame category of each target detection frame are lower than a second preset threshold based on the detection frame category and the detection frame score;

[0138] A coordinate determination unit is configured to determine the coordinates of the filtered target detection frame on the detection image based on the detection frame coordinates.

[0139] For related explanations, please refer to the related descriptions and effects corresponding to the steps in the above method embodiment, and no further details will be given here.

[0140] According to a fourth aspect of the present application, a detection device is provided, the detection device is connected to an embedded platform, the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing of an initial target recognition algorithm model according to a network framework of the embedded platform, the model processing comprising: determining supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupported operators to determine the target recognition algorithm model according to the supported operators and the unsupported operators after replacement processing; the detection device is connected to the embedded platform.

[0141] like Figure 8 As shown, the detection device 80 includes:

[0142] An acquisition unit 81 is configured to acquire a detection image and send the detection image to the embedded platform, so that the embedded platform recognizes target information in the detection image based on a target recognition algorithm model and feeds the target information back to the detection device;

[0143] The marking and display unit 82 is configured to mark the corresponding target object or target pedestrian in the detection image according to the target information, and display the marked detection image on the display interface.

[0144] In one embodiment, the supported operators and unsupported operators corresponding to the network framework in the initial target recognition algorithm model are determined specifically as follows: the initial target recognition algorithm model is parsed to obtain a static calculation graph, which includes all first operators of the initial target recognition algorithm model; all first operators in the static calculation graph are traversed, and each first operator is determined as a supported operator or an unsupported operator according to a preset operator support table of the network framework.

[0145] In one embodiment, a target recognition algorithm model is determined based on supported operators and unsupported operators after replacement processing, specifically: supporting operators and unsupported operators after replacement processing are used as all second operators in a static calculation graph; all second operators in the static calculation graph are traversed to determine the computing resources required for each second operator to be deployed in the network framework; second operators whose computing resources reach a first preset threshold are replaced to obtain all third operators in the static calculation graph, and the target recognition algorithm model is determined based on the third operators.

[0146] In one embodiment, the target recognition algorithm model is determined according to the third operator, specifically: identifying the operator set in the constant layer of the third operator, and deleting the operator set to obtain all fourth operators in the static calculation graph, and determining the target recognition algorithm model according to the fourth operator; or, fusing the third operator of the same type according to the category of the third operator to obtain all fifth operators in the static calculation graph, and determining the target recognition algorithm model according to the fifth operator.

[0147] For related explanations, please refer to the related descriptions and effects corresponding to the steps in the above method embodiment, and no further details will be given here.

[0148] The present application also provides an electronic device, such as Fig. 9 As shown, the electronic device includes: a processor 91, and a memory 92 communicatively connected to the processor 91;

[0149] The memory 91 stores computer-executable instructions;

[0150] The processor 92 executes the computer execution instructions stored in the memory 91 , so that the electronic device executes any one of the target detection methods based on the embedded platform provided in the above embodiments, wherein the memory 92 and the processor 91 are connected via a bus 93 .

[0151] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the steps in the above method embodiment, and no further details will be given here.

[0152] An embodiment of the present application also provides a chip, including a memory and a processor, the memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory to execute any one of the target detection methods based on the embedded platform provided in the above embodiments.

[0153] For related explanations, please refer to the related descriptions and effects corresponding to the steps in the above method embodiment, and no further details will be given here.

[0154] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the target detection method based on an embedded platform provided in an embodiment of the present application.

[0155] Among them, the computer readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0156] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the target detection methods based on an embedded platform provided in the above embodiments.

[0157] Fig.10 It is a block diagram of a terminal device shown in an exemplary embodiment of the present application. The terminal device 800 can be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0158] The terminal device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0159] The processing component 802 generally controls the overall operation of the terminal device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0160] The memory 804 is configured to store various types of data to support operations on the terminal device 800. Examples of such data include instructions for any application or method operating on the terminal device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0161] The power supply component 806 provides power to various components of the terminal device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device 800.

[0162] The multimedia component 808 includes a screen that provides an output interface between the terminal device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the terminal device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the terminal device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0164] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0165] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the terminal device 800. For example, the sensor assembly 814 can detect the open / closed state of the terminal device 800, the relative positioning of the components, such as the display and keypad of the terminal device 800, and the sensor assembly 814 can also detect the position change of the terminal device 800 or a component of the terminal device 800, the presence or absence of contact between the user and the terminal device 800, the orientation or acceleration / deceleration of the terminal device 800, and the temperature change of the terminal device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0166] The communication component 816 is configured to facilitate wired or wireless communication between the terminal device 800 and other devices. The terminal device 800 can access a wireless network based on a communication standard, such as WiFi, 3G, 4G, 5G or other standard communication networks, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0167] In an exemplary embodiment, the terminal device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the methods provided in the above-mentioned embodiments of the present application.

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of a terminal device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] The embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the terminal device, the terminal device 800 can execute the method provided by the above-mentioned embodiment of the present application.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0171] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0172] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A target detection method based on an embedded platform, characterized in that: The embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing includes: determining supportable operators and unsupportable operators corresponding to the network framework in the initial target recognition algorithm model, and replacing the unsupportable operators to determine the target recognition algorithm model according to the supportable operators and the unsupportable operators after the replacement processing; The method is applied to the embedded platform. For each group of detection devices, the embedded platform is embedded in a first detection device in the group of detection devices and connected to at least one other second detection device in the group of detection devices. The method includes: Acquire detection images collected by a first detection device and at least one second detection device; Performing image preprocessing on the detection image based on a graphics card, wherein the image preprocessing includes at least one of the following: image scaling, color filling and color conversion; Converting the detected image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supportable operator and the unsupportable operator after replacement processing in the target recognition algorithm model; Inputting the target input data into the target recognition algorithm model to identify target information in the detection image, wherein the target information includes target object information or target pedestrian information; the target information carries target detection frame information; A target algorithm is used to filter out a part of the target detection frame according to the target detection frame information to obtain processed target information; the target algorithm is different from the target recognition algorithm model; Feeding back the processed target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the processed target information, and displays the marked detection image on the display interface; The adopting target algorithm to filter out a part of the target detection frame according to the target detection frame information to obtain processed target information includes: Parsing the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame; Based on the detection frame category and the detection frame score, filter out target detection frames whose detection frame scores in the corresponding detection frame category of each target detection frame are lower than a second preset threshold; Determine the coordinates of the filtered target detection frame on the detection image to obtain processed target information; wherein, the initial target recognition algorithm model is subjected to model processing to obtain the target recognition algorithm model, including: Parsing the initial target recognition algorithm model to obtain a static calculation graph, wherein the static calculation graph includes all first operators of the initial target recognition algorithm model; Traversing all first operators in the static computation graph, and determining each first operator as a supported operator or an unsupported operator according to a preset operator support table of the network framework; using the supported operators and the replaced unsupported operators as all second operators in the static computation graph; Traversing all second operators in the static computation graph, and determining computing resources required for each second operator to be deployed in the network framework; Replace the second operator whose computing resources reach the first preset threshold value to obtain all the third operators in the static computing graph Identify a set of operators in the constant layer in the third operator, delete the set of operators, obtain all fourth operators in the static calculation graph, and determine the target recognition algorithm model according to the fourth operators; or, The third operator is subjected to similar operator fusion according to the category of the third operator to obtain all fifth operators in the static calculation graph, and the target recognition algorithm model is determined according to the fifth operator.

2. A target detection method based on an embedded platform, characterized in that: The embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on the initial target recognition algorithm model according to the network framework of the embedded platform, and the model processing includes: parsing the initial target recognition algorithm model to obtain a static calculation graph, wherein the static calculation graph includes all first operators of the initial target recognition algorithm model; traversing all first operators in the static calculation graph, and determining each first operator as a supportable operator or an unsupportable operator according to a preset operator support table of the network framework; using the supportable operator and the unsupportable operator after replacement processing as all second operators in the static calculation graph. ; Traverse all the second operators in the static calculation graph, and determine the computing resources required for each of the second operators to be deployed in the network framework; replace the second operators whose computing resources reach the first preset threshold value to obtain all the third operators in the static calculation graph; identify the operator set in the constant layer of the third operator, and delete the operator set to obtain all the fourth operators in the static calculation graph, and determine the target recognition algorithm model according to the fourth operator; or, according to the category of the third operator, fuse the third operator with operators of the same type to obtain all the fifth operators in the static calculation graph, and determine the target recognition algorithm model according to the fifth operator; The method is applied to a detection device, wherein the detection device comprises a plurality of groups, wherein a first detection device in each group of detection devices is embedded with the embedded platform, and at least one remaining second detection device in the same group is connected to the embedded platform in which the first detection device is embedded; the method comprises: Acquire a detection image, and send the detection image to the embedded platform, so that the embedded platform performs image preprocessing on the detection image based on the graphics card, wherein the image preprocessing includes at least one of the following: image proportional scaling, color filling and color conversion; convert the detection image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supportable operator and the unsupportable operator after the replacement processing in the target recognition algorithm model; identify the target information carrying the target detection frame information in the detection image based on the target recognition algorithm model, and use the target algorithm to parse the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame, filter out the target detection frames whose detection frame scores in the corresponding detection frame category of each target detection frame are lower than a second preset threshold based on the detection frame category and the detection frame score, determine the coordinates of the filtered target detection frame on the detection image, obtain the processed target information, and feed the processed target information back to the detection device; The corresponding target object or target pedestrian is marked in the detection image according to the target information, and the marked detection image is displayed on the display interface.

3. An embedded platform, characterized in that the embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on an initial target recognition algorithm model according to a network framework of the embedded platform, and the model processing includes: Parsing the initial target recognition algorithm model to obtain a static calculation graph, wherein the static calculation graph includes all first operators of the initial target recognition algorithm model; Traversing all first operators in the static computation graph, and determining each first operator as a supported operator or an unsupported operator according to a preset operator support table of the network framework; The supported operators and the unsupported operators after replacement are used as all the second operators in the static calculation graph; all the second operators in the static calculation graph are traversed to determine the computing resources required for each second operator to be deployed in the network framework; the second operators whose computing resources reach a first preset threshold are replaced to obtain all the third operators in the static calculation graph; the operator set in the constant layer of the third operators is identified, and the operator set is deleted to obtain all the fourth operators in the static calculation graph, and the target recognition algorithm model is determined according to the fourth operators; or, the third operators are fused with operators of the same type according to the category of the third operators to obtain all the fifth operators in the static calculation graph, and the target recognition algorithm model is determined according to the fifth operators; The embedded platform is connected to the detection device, and the embedded platform includes: An acquisition unit, configured to acquire a detection image collected by the detection device; An identification unit, configured to identify target information in the detection image based on the target recognition algorithm model, wherein the target information includes target object information or target pedestrian information; and the target information carries target detection frame information; A processing unit, which is configured to use a target algorithm to filter out a part of the target detection frame according to the target detection frame information to obtain processed target information; the target algorithm is different from the target recognition algorithm model; a transmission unit configured to feed back the processed target information to the detection device, so that the detection device marks the corresponding target object or target pedestrian in the detection image according to the processed target information, and displays the marked detection image on a display interface; The processing unit includes: a parsing unit, a filtering unit and a coordinate determination unit; The parsing unit is configured to parse the target detection frame information to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame; The filtering unit is configured to filter out target detection frames whose detection frame scores in the corresponding detection frame categories of the target detection frames are lower than a second preset threshold based on the detection frame categories and the detection frame scores; The coordinate determination unit is configured to determine the coordinates of the filtered target detection frame on the detection image to obtain processed target information.

4. A detection device, characterized in that: The detection device comprises a plurality of groups, wherein a first detection device in each group of detection devices is embedded with an embedded platform, and at least one remaining second detection device in the same group is connected to the embedded platform embedded with the first detection device; The embedded platform deploys a target recognition algorithm model, wherein the target recognition algorithm model is deployed after model processing is performed on the initial target recognition algorithm model according to the network framework of the embedded platform, and the model processing includes: parsing the initial target recognition algorithm model to obtain a static calculation graph, wherein the static calculation graph includes all first operators of the initial target recognition algorithm model; traversing all first operators in the static calculation graph, and determining each first operator as a supportable operator or an unsupportable operator according to a preset operator support table of the network framework; using the supportable operator and the unsupportable operator after replacement processing as all second operators in the static calculation graph. ; Traverse all the second operators in the static calculation graph, and determine the computing resources required for each of the second operators to be deployed in the network framework; replace the second operators whose computing resources reach the first preset threshold value to obtain all the third operators in the static calculation graph; identify the operator set in the constant layer of the third operator, and delete the operator set to obtain all the fourth operators in the static calculation graph, and determine the target recognition algorithm model according to the fourth operator; or, according to the category of the third operator, fuse the third operator with operators of the same type to obtain all the fifth operators in the static calculation graph, and determine the target recognition algorithm model according to the fifth operator; The detection device is connected to the embedded platform, and the detection device includes: A collection unit, which is configured to collect a detection image and send the detection image to the embedded platform, so that the embedded platform performs image preprocessing on the detection image based on a graphics card, wherein the image preprocessing includes at least one of the following: image proportional scaling, color filling and color conversion; converting the detection image after image preprocessing into target input data corresponding to the target recognition algorithm model; the target input data is the actual parameters corresponding to the supportable operator and the unsupportable operator after the replacement processing in the target recognition algorithm model; identifying the target information carrying the target detection frame information in the detection image based on the target recognition algorithm model, and parsing the target detection frame information using a target algorithm to obtain the detection frame coordinates, detection frame category and detection frame score of each target detection frame, filtering out the target detection frames whose detection frame scores in the corresponding detection frame category of each target detection frame are lower than a second preset threshold based on the detection frame category and the detection frame score, determining the coordinates of the filtered target detection frame on the detection image, obtaining the processed target information, and feeding back the processed target information to the detection device; the target algorithm is different from the target recognition algorithm model; A marking and display unit is configured to mark the corresponding target object or target pedestrian in the detection image according to the target information, and display the marked detection image on a display interface.

5. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the target detection method based on the embedded platform described in claim 1, or the target detection method based on the embedded platform described in claim 2.