Target identification method suitable for mobile terminal, plug-in, equipment and storage medium

Abstract

The invention provides a target identification method suitable for a mobile terminal, a plug-in, equipment and a storage medium, and the method comprises the steps: loading a trained target detection model into a memory of the mobile terminal, and obtaining a target reference library; transmitting a mobile terminal camera module handle to a camera driving layer of a mobile terminal operating system to open a camera; performing data preprocessing on metadata acquired by a camera of the mobile terminal to obtain optimized image matrix data; analyzing a target contained in the picture frame based on the optimized image matrix data by using a target reference library to obtain an identification result; and returning the identification result to the upper application of the mobile terminal and turning off the camera. The problem that the mobile terminal cannot directly apply the target detection method is solved, it is ensured that the target detection method can adapt to and efficiently run on the mobile terminal equipment, the dependence on a computer is eliminated, and real-time and accurate target detection and recognition can be carried out even in the environment without the computer.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile terminal target recognition, and in particular to a target recognition method, plug-in, device and storage medium suitable for a mobile terminal. Background Art

[0002] At present, most object detection technologies are mainly concentrated on computer applications. The reason behind this phenomenon is that computer devices have more powerful central processing units (CPUs) and graphics processing units (GPUs) with ultra-high computing power compared to mobile devices. This hardware advantage allows researchers and developers to devote more energy to using computer devices to train and deploy object detection algorithms to achieve the recognition and detection of objects in materials such as images, videos, and audio. For example, the Yolo (You Only Look Once) series of models can quickly identify objects in materials on the computer side, showing excellent detection performance.

[0003] However, this method of relying on the computer for target detection has obvious limitations. First, since the target detection process must rely on the computer device, it is impossible to perform target detection anytime and anywhere in the actual production environment, which limits its scope of application. Secondly, although the existing target detection algorithm performs well on the computer, its actual application process requires collecting materials through mobile devices, and then transferring these materials to the computer for analysis and processing, and finally transmitting the analysis results back to the mobile device. This process is not only time-consuming, but also has obvious deficiencies in real-time performance, which greatly reduces the user experience. Therefore, although the target detection method on the computer performs well in some aspects, it is not the best solution in actual applications because it cannot achieve fast and real-time target detection and recognition. Summary of the invention

[0004] In view of the above problems, an embodiment of the present invention provides a target recognition method, plug-in, device and storage medium applicable to a mobile terminal, which solves the existing technical problems.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a target recognition method applicable to a mobile terminal, the method comprising:

[0007] Load the trained object detection model into the memory of the mobile terminal to obtain the object benchmark library;

[0008] Pass the mobile camera module handle to the camera driver layer of the mobile operating system to open the camera;

[0009] Perform data preprocessing on the metadata collected by the mobile camera to obtain optimized image matrix data;

[0010] Analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain the recognition result;

[0011] The recognition result is sent back to the upper-layer application on the mobile terminal and the camera is closed.

[0012] In one embodiment, the step of loading the trained target detection model into the memory of the mobile terminal to obtain the target reference library includes:

[0013] Install the neural network forward reasoning framework on the mobile terminal;

[0014] Read the binary file of the trained object detection model to obtain the model file;

[0015] Read the parameter configuration of the trained target detection model to obtain the parameter file;

[0016] Call the loadModel function to determine the access method and storage path of the model file and parameter file;

[0017] Call the jni function to pass the access mode and storage path to the application layer;

[0018] After obtaining the access mode and storage path passed by the application layer, the neural network forward reasoning framework calls the load_paeam function to load the parameter file and model file into the memory of the mobile terminal to obtain the target benchmark library.

[0019] In one embodiment, a method for determining whether the target detection model is loaded successfully is also included:

[0020] Open the resource file of the storage path through AssetManager and perform the read operation;

[0021] If the resource file is successfully opened and read without error and a non-zero value is returned, it means that the target detection model is loaded successfully;

[0022] If any error occurs during the process of opening or reading the resource file and the return value is 0, it means that the object detection model failed to load.

[0023] In one embodiment, the transferring of the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera includes:

[0024] Pass the mobile phone camera module handle to the jni layer through the setOutputWindow function;

[0025] Call the camera driver through the openCamera function to open the mobile camera.

[0026] In one embodiment, the performing data preprocessing on the metadata collected by the camera of the mobile terminal to obtain optimized image matrix data includes:

[0027] Configure the metadata collected by the mobile camera to NV21 format;

[0028] Create an OpenCV Mat object, configure the height of the OpenCV Mat object to be 1.5 times the height of the image in the NV21 format metadata, configure the width of the OpenCV Mat object to be the width of the image in the NV21 format metadata, and configure an 8-bit unsigned single channel to store the components in the NV21 format metadata as single-channel data sequentially;

[0029] Call the put function of the OpenCV Mat object to fill the NV21 format metadata into the created OpenCV Mat object to obtain the OpenCV Mat matrix data;

[0030] The OpenCV Mat matrix data is imported into OpenCV to perform frame data preprocessing to obtain optimized image matrix data; the frame data preprocessing includes grayscale processing, rotation processing and sharpening processing.

[0031] In one embodiment, the use of the target reference library to analyze the target contained in the picture frame based on the optimized image matrix data to obtain the recognition result includes:

[0032] Configure the data conversion function to convert the optimized image matrix data into the inference image data matrix;

[0033] Import the inference image data matrix into the target benchmark library, use the convolution layer and corresponding activation function of the target benchmark library to extract local features in the inference image data matrix, and obtain a feature map;

[0034] Use the pooling layer and corresponding activation function of the target benchmark library to reduce the spatial dimension of the feature map and obtain the pooled feature map;

[0035] The fully connected layer of the target benchmark library is used to perform target recognition on the pooled feature map, and then the output layer configured with the optimization function is input to obtain the recognition result.

[0036] In one embodiment, transmitting the recognition result back to the upper layer application of the mobile terminal and closing the camera includes:

[0037] Render the recognition result to obtain rendering data;

[0038] Call the mobile application extension result to upload the rendering data to the upper-layer application on the mobile terminal;

[0039] After the rendering data is uploaded successfully, the camera driver is called through the closeCamera function to close the mobile camera.

[0040] In a second aspect, the present invention provides a target recognition plug-in applicable to a mobile terminal, wherein the target recognition plug-in is obtained by cross-compilation, and the target recognition plug-in includes:

[0041] Loading module: used to load the trained target detection model into the memory of the mobile terminal to obtain the target benchmark library;

[0042] Transfer module: used to transfer the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera;

[0043] Conversion module: used to pre-process the metadata collected by the mobile camera to obtain optimized image matrix data;

[0044] Analysis module: used to analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain the recognition result;

[0045] Feedback module: used to transmit the recognition results back to the upper-layer application on the mobile terminal and turn off the camera.

[0046] In one embodiment, the loading module is specifically used for:

[0047] Install the neural network forward reasoning framework on the mobile terminal;

[0048] Read the binary file of the trained object detection model to obtain the model file;

[0049] Read the parameter configuration of the trained target detection model to obtain the parameter file;

[0050] Call the loadModel function to determine the access method and storage path of the model file and parameter file;

[0051] Call the jni function to pass the access mode and storage path to the application layer;

[0052] After obtaining the access mode and storage path passed by the application layer, the neural network forward reasoning framework calls the load_paeam function to load the parameter file and model file into the memory of the mobile terminal to obtain the target benchmark library.

[0053] In one embodiment, the loading module also includes a method for determining whether the target detection model is loaded successfully:

[0054] Open the resource file of the storage path through AssetManager and perform the read operation;

[0055] If the resource file is successfully opened and read without error and a non-zero value is returned, it means that the target detection model is loaded successfully;

[0056] If any error occurs during the process of opening or reading the resource file and the return value is 0, it means that the object detection model failed to load.

[0057] In one embodiment, the transmission module is specifically used for:

[0058] Pass the mobile phone camera module handle to the jni layer through the setOutputWindow function;

[0059] Call the camera driver through the openCamera function to open the mobile camera.

[0060] In one embodiment, the conversion module is specifically used for:

[0061] Configure the metadata collected by the mobile camera to NV21 format;

[0062] Create an OpenCV Mat object, configure the height of the OpenCV Mat object to be 1.5 times the height of the image in the NV21 format metadata, configure the width of the OpenCV Mat object to be the width of the image in the NV21 format metadata, and configure an 8-bit unsigned single channel to store the components in the NV21 format metadata as single-channel data sequentially;

[0063] Call the put function of the OpenCV Mat object to fill the NV21 format metadata into the created OpenCV Mat object to obtain the OpenCV Mat matrix data;

[0064] The OpenCV Mat matrix data is imported into OpenCV to perform frame data preprocessing to obtain optimized image matrix data; the frame data preprocessing includes grayscale processing, rotation processing and sharpening processing.

[0065] In one embodiment, the analysis module is specifically used for:

[0066] Configure the data conversion function to convert the optimized image matrix data into the inference image data matrix;

[0067] Import the inference image data matrix into the target benchmark library, use the convolution layer and corresponding activation function of the target benchmark library to extract local features in the inference image data matrix, and obtain a feature map;

[0068] Use the pooling layer and corresponding activation function of the target benchmark library to reduce the spatial dimension of the feature map and obtain the pooled feature map;

[0069] The fully connected layer of the target benchmark library is used to perform target recognition on the pooled feature map, and then the output layer configured with the optimization function is input to obtain the recognition result.

[0070] In one embodiment, the feedback module is specifically used for:

[0071] Render the recognition result to obtain rendering data;

[0072] Call the mobile application extension result to upload the rendering data to the upper-layer application on the mobile terminal;

[0073] After the rendering data is uploaded successfully, the camera driver is called through the closeCamera function to close the mobile camera.

[0074] In a third aspect, the present invention provides an electronic device, comprising:

[0075] processor, memory, and an interface for communicating with a gateway;

[0076] The memory is used to store programs and data, and the processor calls the program stored in the memory to execute a target recognition method suitable for a mobile terminal provided in any one of the first aspects.

[0077] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a program, and when the program is executed by a processor, the program is used to execute a target recognition method applicable to a mobile terminal provided in any one of the first aspects.

[0078] From the above description, it can be seen that the embodiments of the present invention provide a target recognition method, plug-in, device and storage medium suitable for mobile terminals, which successfully solves the problem that the mobile terminal cannot directly apply the target recognition method, so that the target recognition process no longer needs to rely on computer devices. This improvement allows users to perform target recognition anytime and anywhere in an actual production environment, thereby greatly broadening the scope of application of the target recognition method. At the same time, it also makes up for the previous shortcomings in real-time performance and significantly improves the user experience. It realizes that the target recognition method can be adapted and efficiently run on mobile devices, completely getting rid of the dependence on computers, and even in an environment without a computer, users can perform real-time and accurate target recognition operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 The figure is a flow chart of a target recognition method applicable to a mobile terminal provided by an embodiment of the present invention;

[0080] Figure 2 The figure is a schematic diagram of the structure of a target recognition plug-in applicable to a mobile terminal provided by an embodiment of the present invention;

[0081] Figure 3FIG. 1 is a schematic structural diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solution and advantages of the present invention clearer and more understandable, the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0083] Based on the shortcomings of the prior art, the present invention provides a specific implementation of a target recognition method suitable for a mobile terminal, such as Figure 1 As shown, the method includes:

[0084] S110: Load the trained target detection model into the memory of the mobile terminal to obtain a target benchmark library;

[0085] The purpose of this step is to build a target reference library on the mobile terminal. Mobile terminals include devices with camera functions, such as smart phones, tablet computers, smart glasses, etc. When training the target detection model, first, conduct on-site data collection, clean the collected data, discard some pictures with small markers, and samples with unclear image quality, and then use the annotation tool to annotate the image, annotate the objects in each picture, and record the categories at the same time, generate the relevant files of the marking box, that is, the supporting files of the data set, and then obtain the data sample library. Then the data sample library is input into the backbone network of the target detection model, and after multiple convolution and normalization operations, as well as image slicing and feature extraction operations, the distribution of the data is learned and the image features are extracted to obtain three feature layers of different sizes, and then the three feature maps generated after processing are input into the output module. The output module generates a bounding box and confidence according to the three feature maps, and uses the optimization and update algorithm of the output end to make each object finally obtain an optimal target box and the highest confidence, completing the training of the target detection model.

[0086] Loading the trained object detection model into the memory of the mobile terminal to obtain the target benchmark library includes the following steps:

[0087] The neural network forward reasoning framework is installed on the mobile terminal; the neural network forward reasoning framework ncnn supports the import of multiple deep learning models, so that developers can easily deploy the trained models to mobile devices. In this embodiment, the target detection model preferably uses the Yolov8 model. Of course, various versions of Yolo models can also be selected according to actual needs.

[0088] Read the binary file of the trained target detection model to get the model file; the model file *bin is the trained yolo model binary file;

[0089] Read the parameter configuration of the trained target detection model to get the parameter file; the parameter file *.param is the detection parameter configuration of the yolo model;

[0090] Call the loadModel function to determine the access method and storage path of the model file and parameter file;

[0091] Call the jni function to pass the access mode and storage path to the application layer (i.e., the C++ code layer);

[0092] After obtaining the access mode and storage path passed by the application layer, the neural network forward reasoning framework calls the load_paeam function to load the parameter file and model file into the memory of the mobile terminal to obtain the target benchmark library.

[0093] It also includes a method to determine whether the target detection model has been loaded successfully:

[0094] Open the resource file of the storage path through AssetManager and perform the read operation;

[0095] If the resource file is successfully opened and read without error and a non-zero value is returned, it means that the target detection model is loaded successfully;

[0096] If any error occurs during the process of opening or reading the resource file and the return value is 0, it means that the target detection model fails to load. If the target detection model fails to load, check the storage location of the model file and parameter file.

[0097] After the target detection model is loaded successfully, the target detection model loaded into the mobile terminal memory can be used as the target reference library. In this way, the frame data captured by the mobile terminal camera can be compared with the target reference library. In layman's terms, when the camera captures a frame, the target reference library will analyze it according to the trained target detection model to identify the data in the frame, and finally return the matching result to the mobile terminal graphical user interface, so that the user can get the recognition result intuitively.

[0098] S120: passing the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera;

[0099] The purpose of this step is to obtain the mobile camera permission and then obtain the image data.

[0100] The specific implementation method includes the following steps:

[0101] Pass the mobile phone camera module handle to the jni layer through the setOutputWindow function;

[0102] Call the camera driver through the openCamera function to open the mobile camera.

[0103] This method can be used to open the mobile camera at the jni layer. Compared with opening the camera at the application layer, it avoids conversion through third-party libraries or frameworks, reduces the overhead of data transmission, and thus improves performance. It also avoids the extra memory usage generated during image data transmission, which can free up more memory space for the target detection model. It can maximize performance and provide more memory space for running the target detection model on the mobile terminal, which helps to improve real-time performance, ensure stable operation, and give full play to the potential of mobile hardware.

[0104] S130: performing data preprocessing on the metadata collected by the camera of the mobile terminal to obtain optimized image matrix data;

[0105] The purpose of this step is to obtain image data that is more conducive to target detection model to identify the target, thereby improving the efficiency and accuracy of the target detection model. The specific implementation method includes the following steps:

[0106] The metadata collected by the mobile camera is configured in NV21 format. Image metadata encoded in NV21 format can compress video data for transmission to reduce the amount of data.

[0107] Create an OpenCV Mat object, configure the height of the OpenCV Mat object to be 1.5 times the image height in the NV21 format metadata, configure the width of the OpenCV Mat object to be the width of the image in the NV21 format metadata, and configure an 8-bit unsigned single channel to store the components in the NV21 format metadata as single-channel data in sequence; using single-channel storage, there is no need to distinguish between Y and UV components during processing, which simplifies the complexity of data access and processing; single-channel data occupies less memory space than dual-channel data, which helps save resources; sequentially stored data can be cached and accessed more efficiently, improving access speed; when transmitting data, the single-channel data format can reduce the overhead of data packaging and unpacking, thereby improving transmission efficiency; the single-channel data format is more easily compatible with different image processing libraries and frameworks, and the single-channel data format can reduce the complexity of the target detection model, making the target detection model implementation more direct and efficient;

[0108] Call the put function of the OpenCV Mat object to fill the NV21 format metadata into the created OpenCV Mat object to obtain the OpenCV Mat matrix data;

[0109] Import the OpenCV Mat matrix data into OpenCV for frame data preprocessing to obtain optimized image matrix data; frame data preprocessing includes grayscale processing, rotation processing and sharpening processing. Grayscale processing converts color images into grayscale images to reduce computational complexity; rotation processing ensures that the image direction is correct and adapts to the input image direction of the target detection model; sharpening processing enhances the edge details of the image and improves the target detection model's ability to recognize target features.

[0110] S140: Analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain a recognition result;

[0111] The purpose of this step is to identify the target in the image taken by the mobile camera based on the target reference library, and then obtain the recognition result. The specific implementation method includes the following steps:

[0112] Configure the data conversion function to convert the optimized image matrix data into the inference image data matrix; the function of the data conversion function is to convert the optimized image matrix data into an image data matrix suitable for inference, that is, to convert the OpenCV Mat matrix data into ncnn Mat matrix data (inference image data matrix, used for target benchmark library inference recognition), the function is: ncnn::Mat in = ncnn::Mat::from_pixels(rgb.data,ncnn::Mat::PIXEL_RGB,width,height); where rgb.data is the opencv Mat matrix data;

[0113] Import the inference image data matrix into the target benchmark library, use the convolution layer and corresponding activation function of the target benchmark library to extract local features in the inference image data matrix, obtain feature maps, and capture key information in the image;

[0114] Use the pooling layer and corresponding activation function of the target benchmark library to reduce the spatial dimension of the feature map to obtain a pooled feature map, reduce the amount of data in the feature map, and retain important feature information;

[0115] The pooled feature map is used to identify the target using the fully connected layer of the target benchmark library, and then input into the output layer configured with an optimization function (such as a softmax function) to obtain the recognition result.

[0116] S150: Send the recognition result back to the upper-layer application on the mobile terminal and close the camera.

[0117] Render the recognition results to obtain rendering data; for example, draw a border around the recognized target, indicate the probability distribution of the target in the image by the depth of color, perform pixel-level segmentation on each target instance, draw marker points on the key parts of the target, add text description around the target, etc.

[0118] Call the mobile application extension result to upload the rendering data to the upper-layer application on the mobile terminal;

[0119] After the rendering data is uploaded successfully, the camera driver is called through the closeCamera function to close the mobile camera to ensure that the camera resources are properly managed and released.

[0120] In summary, the method of the present invention successfully solves the problem that the target recognition method cannot be directly applied on the mobile terminal, so that the target recognition process no longer needs to rely on the computer device. This improvement allows users to perform target recognition anytime and anywhere in an actual production environment, thereby greatly broadening the scope of application of the target recognition method. At the same time, it also makes up for the previous shortcomings in real-time performance and significantly improves the user experience. It realizes that the target recognition method can be adapted and efficiently run on mobile devices, completely getting rid of the dependence on computers, and even in an environment without a computer, users can perform real-time and accurate target recognition operations.

[0121] Based on the same inventive concept, the embodiment of the present application also provides a target recognition plug-in plug-in applicable to a mobile terminal, which can be used to implement the target recognition plug-in method applicable to a mobile terminal described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the target recognition plug-in plug-in applicable to a mobile terminal is similar to that of the target recognition plug-in method applicable to a mobile terminal, the implementation of the target recognition plug-in plug-in applicable to a mobile terminal can refer to the implementation of the target recognition plug-in method applicable to a mobile terminal, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0122] The present invention provides a target recognition plug-in suitable for a mobile terminal, such as Figure 2 As shown, in Figure 2The target recognition plug-in is obtained by cross-compilation, and the plug-in packaging is completed through CMake. CMake (Cross-platform Make) is an open source, cross-platform automated build system. In the project root directory, you can use the following command to create a directory named build: cmake-B build-S. Here, the -B build option specifies the name of the build directory, and -S. specifies the source code directory, that is, the current directory. Once the build directory is created and the build file is generated, enter the directory and use the following command to compile the project: cmake--build. Finally, you can find a plug-in named libyolov8ncnn.so in the code project app / build / intermediates / cmake / debug / obj / arm64-v8a / directory. Among them, arm64-v8a is the CPU architecture supported by the current mobile terminal. This directory may vary for different mobile phones. The target recognition plug-in includes:

[0123] Loading module 210: used to load the trained target detection model into the memory of the mobile terminal to obtain a target reference library;

[0124] The transfer module 220 is used to transfer the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera;

[0125] Conversion module 230: used to perform data preprocessing on the metadata collected by the camera of the mobile terminal to obtain optimized image matrix data;

[0126] Analysis module 240: used to analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain a recognition result;

[0127] Feedback module 250: used to transmit the recognition result back to the upper layer application of the mobile terminal and turn off the camera.

[0128] In one embodiment of the present invention, the loading module 210 is specifically used for:

[0129] Install the neural network forward reasoning framework on the mobile terminal;

[0130] Read the binary file of the trained object detection model to obtain the model file;

[0131] Read the parameter configuration of the trained target detection model to obtain the parameter file;

[0132] Call the loadModel function to determine the access method and storage path of the model file and parameter file;

[0133] Call the jni function to pass the access mode and storage path to the application layer;

[0134] After obtaining the access mode and storage path passed by the application layer, the neural network forward reasoning framework calls the load_paeam function to load the parameter file and model file into the memory of the mobile terminal to obtain the target benchmark library.

[0135] In one embodiment of the present invention, the loading module 210 also includes a method for determining whether the target detection model is loaded successfully:

[0136] Open the resource file of the storage path through AssetManager and perform the read operation;

[0137] If the resource file is successfully opened and read without error and a non-zero value is returned, it means that the target detection model is loaded successfully;

[0138] If any error occurs during the process of opening or reading the resource file and the return value is 0, it means that the object detection model failed to load.

[0139] In one embodiment of the present invention, the transmission module 220 is specifically used for:

[0140] Pass the mobile phone camera module handle to the jni layer through the setOutputWindow function;

[0141] Call the camera driver through the openCamera function to open the mobile camera.

[0142] In one embodiment of the present invention, the conversion module 230 is specifically used for:

[0143] Configure the metadata collected by the mobile camera to NV21 format;

[0144] Create an OpenCV Mat object, configure the height of the OpenCV Mat object to be 1.5 times the height of the image in the NV21 format metadata, configure the width of the OpenCV Mat object to be the width of the image in the NV21 format metadata, and configure an 8-bit unsigned single channel to store the components in the NV21 format metadata as single-channel data sequentially;

[0145] Call the put function of the OpenCV Mat object to fill the NV21 format metadata into the created OpenCV Mat object to obtain the OpenCV Mat matrix data;

[0146] Import the OpenCV Mat matrix data into OpenCV for frame data preprocessing to obtain optimized image matrix data; frame data preprocessing includes grayscale processing, rotation processing and sharpening processing.

[0147] In one embodiment of the present invention, the analysis module 240 is specifically used for:

[0148] Configure the data conversion function to convert the optimized image matrix data into the inference image data matrix;

[0149] Import the inference image data matrix into the target benchmark library, use the convolution layer and corresponding activation function of the target benchmark library to extract local features in the inference image data matrix, and obtain a feature map;

[0150] Use the pooling layer and corresponding activation function of the target benchmark library to reduce the spatial dimension of the feature map and obtain the pooled feature map;

[0151] The pooled feature map is used to identify the target using the fully connected layer of the target benchmark library, and then input into the output layer configured with the optimization function to obtain the recognition result.

[0152] In one embodiment of the present invention, the feedback module 250 is specifically used for:

[0153] Render the recognition result to obtain rendering data;

[0154] Call the mobile application extension result to upload the rendering data to the upper-layer application on the mobile terminal;

[0155] After the rendering data is uploaded successfully, the camera driver is called through the closeCamera function to close the mobile camera.

[0156] The embodiments of the present application also provide a specific implementation of an electronic device that can implement all the steps in the method in the above embodiments, see Figure 3 , the electronic device 300 specifically includes the following contents:

[0157] Processor 310, memory 320, communication unit 330 and bus 340;

[0158] The processor 310 , the memory 320 , and the communication unit 330 communicate with each other via the bus 340 ; the communication unit 330 is used to implement information transmission between server-side devices and terminal devices and other related devices.

[0159] The processor 310 is used to call the computer program in the memory 320. When the processor executes the computer program, all the steps in the target identification plug-in method applicable to the mobile terminal in the above embodiment are implemented.

[0160] Those skilled in the art should understand that the memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store programs, and the processor executes the programs after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0161] The processor may be an integrated circuit chip having the ability to process signals. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0162] The present application also provides a computer-readable storage medium, which includes a program. When the program is executed by a processor, it is used to execute the target recognition plug-in method applicable to a mobile terminal provided by any of the aforementioned method embodiments.

[0163] Those skilled in the art should understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes, and the specific media type is not limited in this application.

[0164] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A target recognition method suitable for a mobile terminal, characterized in that: The method comprises: Load the trained object detection model into the memory of the mobile terminal to obtain the object benchmark library; Pass the mobile camera module handle to the camera driver layer of the mobile operating system to open the camera; Perform data preprocessing on the metadata collected by the mobile camera to obtain optimized image matrix data; Analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain the recognition result; The recognition result is sent back to the upper-layer application on the mobile terminal and the camera is closed.

2. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: The trained target detection model is loaded into the memory of the mobile terminal to obtain the target benchmark library, including: Install the neural network forward reasoning framework on the mobile terminal; Read the binary file of the trained object detection model to obtain the model file; Read the parameter configuration of the trained target detection model to obtain the parameter file; Call the loadModel function to determine the access method and storage path of the model file and parameter file; Call the jni function to pass the access mode and storage path to the application layer; After obtaining the access mode and storage path passed by the application layer, the neural network forward reasoning framework calls the load_paeam function to load the parameter file and model file into the memory of the mobile terminal to obtain the target benchmark library.

3. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: It also includes a method to determine whether the target detection model has been loaded successfully: Open the resource file of the storage path through AssetManager and perform the read operation; If the resource file is successfully opened and read without error and a non-zero value is returned, it means that the target detection model is loaded successfully; If any error occurs during the process of opening or reading the resource file and the return value is 0, it means that the object detection model failed to load.

4. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: The step of passing the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera includes: Pass the mobile phone camera module handle to the jni layer through the setOutputWindow function; Call the camera driver through the openCamera function to open the mobile camera.

5. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: The data preprocessing of the metadata collected by the camera of the mobile terminal to obtain optimized image matrix data includes: Configure the metadata collected by the mobile camera to NV21 format; Create an OpenCV Mat object, configure the height of the OpenCV Mat object to be 1.5 times the height of the image in the NV21 format metadata, configure the width of the OpenCV Mat object to be the width of the image in the NV21 format metadata, and configure an 8-bit unsigned single channel to store the components in the NV21 format metadata as single-channel data sequentially; Call the put function of the OpenCV Mat object to fill the NV21 format metadata into the created OpenCV Mat object to obtain the OpenCV Mat matrix data; The OpenCV Mat matrix data is imported into OpenCV to perform frame data preprocessing to obtain optimized image matrix data; the frame data preprocessing includes grayscale processing, rotation processing and sharpening processing.

6. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: The target reference library is used to analyze the target contained in the picture frame based on the optimized image matrix data to obtain the recognition result, which includes: Configure the data conversion function to convert the optimized image matrix data into the inference image data matrix; Import the inference image data matrix into the target benchmark library, use the convolution layer and corresponding activation function of the target benchmark library to extract local features in the inference image data matrix, and obtain a feature map; Use the pooling layer and corresponding activation function of the target benchmark library to reduce the spatial dimension of the feature map and obtain the pooled feature map; The fully connected layer of the target benchmark library is used to perform target recognition on the pooled feature map, and then the output layer configured with the optimization function is input to obtain the recognition result.

7. A target recognition method suitable for a mobile terminal as claimed in claim 1, characterized in that: The step of transmitting the recognition result back to the upper layer application on the mobile terminal and closing the camera includes: Render the recognition result to obtain rendering data; Call the mobile application extension result to upload the rendering data to the upper-layer application on the mobile terminal; After the rendering data is uploaded successfully, the camera driver is called through the closeCamera function to close the mobile camera.

8. A target recognition plug-in suitable for a mobile terminal, characterized in that: The target recognition plug-in is obtained by cross-compilation, and the target recognition plug-in includes: Loading module: used to load the trained target detection model into the memory of the mobile terminal to obtain the target benchmark library; Transfer module: used to transfer the mobile terminal camera module handle to the camera driver layer of the mobile terminal operating system to open the camera; Conversion module: used to pre-process the metadata collected by the mobile camera to obtain optimized image matrix data; Analysis module: used to analyze the target contained in the picture frame based on the optimized image matrix data using the target reference library to obtain the recognition result; Feedback module: used to transmit the recognition results back to the upper-layer application on the mobile terminal and turn off the camera.

9. An electronic device, characterized in that: include: processor, memory, and an interface for communicating with a gateway; The memory is used to store programs and data, and the processor calls the program stored in the memory to execute the target recognition method suitable for a mobile terminal as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program, and when the program is executed by a processor, it is used to execute the target recognition method applicable to a mobile terminal as described in any one of claims 1 to 7.

Images