Image detection method, electronic device, and storage medium

Abstract

The application provides an image detection method, an electronic device and a storage medium. The method comprises the following steps: obtaining a neural network model, wherein the neural network model comprises n operators; mapping the neural network model into a single linked list structure, wherein the single linked list structure comprises n nodes; establishing a data pair vector corresponding to the single linked list structure based on an output quantity and an input quantity of each node; determining an operator subgroup in the neural network model based on the data pair vector; inputting a target image into the neural network model, recording input data and output data of the operator subgroup; detecting the target image based on the input data and the output data, and outputting a detection result. The image detection result can be quickly obtained through the application.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image detection method, electronic device and storage medium. Background Technology

[0002] In practical applications, when performing image detection using a neural network model installed in an electronic device, image feature data is transferred between operators. Generally, it's necessary to record the input and output data of each operator, and then perform further detection on the image based on this data. However, some complex neural network models have many operators, resulting in a large amount of input and output data to be statistically analyzed. This inevitably leads to excessive memory usage, slow CPU processing, and slow image detection speed in electronic devices. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image detection method, electronic device and storage medium that can quickly output image detection results.

[0004] This application provides an image detection method, the method comprising:

[0005] Obtain a neural network model, wherein the neural network model includes n operators;

[0006] Based on the logical operation relationship between the n operators, the neural network model is mapped to a singly linked list structure, wherein the singly linked list structure includes n nodes;

[0007] Scan each node in the singly linked list structure and count the output and input of each node in the neural network model;

[0008] Based on the output and input of each node, a data pair vector corresponding to the singly linked list structure is established, wherein the data pair vector is [[a1,b1],[a2,b2]…[a...]. i, b i ]…[a n ,b n ]], a i The output of each node, b i The input for each node;

[0009] Based on the data pair vectors, determine the operator subgroups in the neural network model;

[0010] Input the target image into the neural network model, and record the input and output data of the operator subgroup;

[0011] Based on the input and output data, the neural network model is used to detect the target image and output the detection results.

[0012] In one possible implementation, the n nodes in the singly linked list are OP1, OP2, ... OP1. i ,…,OP n Node OP i Output a i Indicates the node OP i The corresponding operator in the neural network model transmits data outwards, and the number of operators receiving the data; the node OP i Input quantity b i Indicates the node OP i The number of other operators that transmit the data when the corresponding operator receives data from other operators in the neural network model.

[0013] In one possible implementation, determining the operator subgroups in the neural network model based on the data pair vectors includes:

[0014] Multiple node subsets are determined based on the data pair vector;

[0015] The operator subgroups in the neural network model are determined based on the plurality of node subsets.

[0016] In one possible implementation, determining multiple subsets of nodes based on the data pair vector includes:

[0017] S51. Traverse the data pairs in the data pair vector starting from [a1,b1];

[0018] S52. Determine whether the data pair meets the push-on-stack condition. If the data pair does not meet the push-on-stack condition, execute S54.

[0019] S53. If the data pair meets the push-to-stack condition, push the data pair onto the stack;

[0020] S54. Determine whether the data pair meets the pop stack condition. If the data pair does not meet the pop stack condition, execute S57.

[0021] S55. If the data pair satisfies the pop stack condition, pop the topmost data pair in the current stack and count the number m of the remaining data pairs in the stack.

[0022] S56. Determine the starting node corresponding to the topmost data pair in the stack, and determine the ending node corresponding to the data pair that satisfies the pop condition. Set the subset consisting of all nodes from the starting node to the ending node as the m+1 level node subset.

[0023] S57. Continue to traverse the data pairs in the data pair vector, repeating S52 to S57 until all data pairs in the data pair vector have been traversed.

[0024] S58. End traversal.

[0025] In one possible implementation, determining the operator subgroups in the neural network model based on the plurality of node subsets includes:

[0026] Traverse the multiple node subsets;

[0027] If no node in any of the multiple node subsets is repeated from any node in the other node subsets, then each node subset is determined as an operator subgroup in the neural network model.

[0028] In one possible implementation, determining the operator subgroups in the neural network model based on the plurality of node subsets further includes:

[0029] Traverse the multiple node subsets to obtain multiple first node subsets and multiple second node subsets, wherein all nodes in the first node subset are contained in the second node subset;

[0030] Remove all nodes from the first node subset within the second node subset;

[0031] The remaining set of nodes in the second node subset is defined as the operator subgroup of the neural network model.

[0032] In one possible implementation, determining the operator subgroups in the neural network model based on the plurality of node subsets further includes:

[0033] The plurality of first node subsets are defined as operator subgroups of the neural network model.

[0034] In one possible implementation, if the data pair [a] i ,b i b in ] i ≥2, determine the data pair [a i ,b i The conditions for popping from the stack are met; if the data pair [a] i ,b i a in ] i ≥2, determine the data pair [a i ,b i The conditions for pushing onto the stack are met.

[0035] This application also provides an electronic device, which includes a processor and a memory, wherein the processor is used to implement the image detection method by executing a computer program stored in the memory.

[0036] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image detection method described above.

[0037] The image detection method, electronic device, and storage medium disclosed in this application can quickly output image detection results. Attached Figure Description

[0038] Figure 1 This is a flowchart of a preferred embodiment of an image detection method disclosed in this application.

[0039] Figure 2 This is a detailed flowchart of step S5 of an image detection method disclosed in this application.

[0040] Figure 3 This is an exemplary singly linked list structure disclosed in this application.

[0041] Figure 4 This is an exemplary node data transmission diagram disclosed in this application.

[0042] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the image detection method of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the image detection method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0045] The image detection method is applied to one or more electronic devices 5. The electronic device 5 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0046] The electronic device 5 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0047] The electronic device 5 may also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0048] The network in which the electronic device 5 is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0049] S1. Obtain a neural network model, wherein the neural network model includes n operators.

[0050] In this embodiment, the neural network model can be a neural network model directly obtained from the network source, or it can be an optimized neural network model obtained from the network source. Optimizing the neural network model can be understood as performing operations such as operator fusion, network pruning, model quantization, and network segmentation. In this embodiment, the neural network model includes n operators, namely op1, op2, ..., opn.

[0051] S2. Based on the logical operation relationship between the n operators, the neural network model is mapped to a singly linked list structure, wherein the singly linked list structure includes n nodes.

[0052] In order to subsequently analyze the input and output data of each operator, the neural network model needs to be mapped to a singly linked list structure.

[0053] In this embodiment, the singly linked list structure includes multiple nodes, and these nodes are arranged in a unidirectional manner. The node at the beginning of the singly linked list structure is the head of the structure, and the node at the end of the structure is the tail. When accessing the singly linked list structure, it is necessary to read from the head to the tail according to the unidirectional order of the linked list.

[0054] In this embodiment, the logical operation relationship between the n operators can be understood as the data transfer relationship between operators. For example, the neural network model includes operator A, operator B, operator C, and operator D. The output data of operator A is the input data of operators B and C, and the output data of operators B and C is the input data of operator D. Thus, the resulting singly linked list structure includes A->B->C->D, or A->C->B->D.

[0055] In this embodiment, the singly linked list structure includes n nodes, where the n nodes are OP1, OP2, ..., OP1. n The n nodes correspond one-to-one with each operator in the neural network model. Based on the dependencies between the n operators, multiple singly linked list structures can be obtained. In this embodiment, one of the singly linked list structures is selected: OP1 → OP2 → ... → OP1. n Further explanation will follow. For example, Figure 3 The structure shown is a singly linked list.

[0056] S3. Scan each node in the singly linked list structure and count the output and input of each node in the neural network model.

[0057] In order to obtain the operator subgroups in the future, it is necessary to count the output and input of each node in the neural network model.

[0058] In this embodiment, the output quantity represents the number of operators that receive the data when the operator corresponding to the node transmits the data outward in the neural network model; the input quantity represents the number of operators that transmit the data when the operator corresponding to the node receives data transmitted by other operators in the neural network model.

[0059] For example, Figure 4 The input of node OP4 is 1, and the output is 4.

[0060] S4. Based on the output and input of each node, establish a data pair vector corresponding to the singly linked list structure, wherein the data pair vector is [[a1,b1],[a2,b2],…,[a...]. i, b i ],…,[a n ,b n ]], a i The output of each node, b i The input for each node.

[0061] In this embodiment, the data pair vector is based on nodes OP1, OP2, ..., and OP. nA two-dimensional array is constructed using the output and input quantities, where [a i, b i ] represents the node OP i The output and input data.

[0062] S5. Determine the operator subgroups in the neural network model based on the data pair vectors.

[0063] In this embodiment, when the electronic device performs image detection using a neural network model, it needs to record the input and output data of each operator. Additionally, the data the electronic device needs to record also includes some invalid data. Thus, the amount of data the electronic device needs to record is enormous, which may lead to excessive memory usage and slow CPU processing. To improve the processing power of the electronic device and quickly obtain image detection results, the method provided in this application can be used to find operator subgroups in the neural network model. In this way, only the input and output data of the operator subgroups need to be recorded before proceeding to the next step of image detection processing.

[0064] Specifically, determining the operator subgroup in the neural network model based on the data pair vector includes: determining multiple node subsets based on the data pair vector; and determining the operator subgroup in the neural network model based on the multiple node subsets.

[0065] See Figure 2 The diagram shown is a detailed flowchart of step S5 of an image detection method disclosed in this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0066] S51. Traverse the data pairs in the data pair vector starting from [a1,b1];

[0067] S52. Determine whether the data pair satisfies the push-on stack condition, wherein satisfying the push-on stack condition means that the data pair [a i ,b i a in ] i ≥2, if the data pair does not meet the push-to-stack condition, execute S54;

[0068] S53. If the data pair meets the push-to-stack condition, push the data pair onto the stack;

[0069] S54. Determine whether the data pair satisfies the pop stack condition, wherein satisfying the pop stack condition means that the data pair [a i ,b i b in ] i ≥2, If the data pair does not meet the pop stack condition, execute S57;

[0070] S55. If the data pair satisfies the pop stack condition, pop the topmost data pair in the current stack and count the number m of the remaining data pairs in the stack.

[0071] S56. Determine the starting node corresponding to the topmost data pair in the stack, and determine the ending node corresponding to the data pair that satisfies the pop stack condition. Set the subset consisting of all nodes from the starting node to the ending node as the m+1 level node subset.

[0072] S57. Continue to traverse the data pairs in the data pair vector, repeating S52 to S57 until all data pairs in the data pair vector have been traversed.

[0073] S58. End traversal.

[0074] After the processing in steps S51 to S58, the resulting subsets of nodes will fall into two categories. One category is that no two subsets contain duplicate nodes; the other category is that one subset is a true subset of the others. The methods for determining the operator subgroups in the neural network model differ depending on the specific circumstances.

[0075] In one embodiment, determining the operator subgroup in the neural network model based on the plurality of node subsets includes: traversing the plurality of node subsets; if any node in any node subset of the plurality of node subsets is not repeated with nodes in other node subsets, determining each node subset as an operator subgroup in the neural network model.

[0076] In another embodiment, determining the operator subgroup in the neural network model based on the plurality of node subsets includes: traversing the plurality of node subsets to obtain a plurality of first node subsets and a plurality of second node subsets, wherein the first node subset is contained in the second node subset; removing all nodes of the first node subset in the second node subset, and taking the remaining set of nodes in the second node subset as the operator subgroup of the neural network model.

[0077] It should be noted that there are no duplicate nodes in the remaining node sets of the first node subset and the second node subset. Therefore, it is necessary to define the multiple first node subsets as operator subgroups of the neural network model.

[0078] For example,

[0079] If the singly linked list structure is OP1->OP2->...->OP 15 Wherein nodes OP1 to OP 11The data pair vectors corresponding to the nodes in the singly linked list are [[2,1],[1,1],[1,2],[4,1],[1,1],[1,1],[1,1],[1,1],[1,1],[1,1],[1,1],[1,1],[1,1],[1,1],[2,1],[1,1],[1,2],[1,4]], respectively, corresponding to operators op1 to op15 of the neural network model.

[0080] Iterate through the data pair vector starting from [2,1];

[0081] Since the data pair [2,1] meets the conditions for being pushed onto the stack, the data pair [2,1] is pushed onto the stack;

[0082] Since the data pair [1,2] satisfies the pop condition of the stack, the topmost data pair [2,1] in the current stack is popped;

[0083] Since the number of data pairs in the stack is 0, nodes OP1, OP2, and OP3 are determined to be a subset of level 1 nodes.

[0084] Continue iterating through the data pair vector;

[0085] Since the data pair [4,1] meets the conditions for being pushed onto the stack, the data pair [4,1] is pushed onto the stack;

[0086] Since the data pair [2,1] meets the conditions for being pushed onto the stack, the data pair [2,1] is pushed onto the stack;

[0087] Since the data pair [1,2] satisfies the pop condition of the stack, the topmost data pair [2,1] in the current stack is popped;

[0088] The number of data pairs in the stack is counted as 1, thus determining the node OP. 12 ,OP 13 ,OP 14 A subset of level 2 nodes;

[0089] Continue iterating through the data pair vector;

[0090] Since the data pair [1,4] satisfies the pop condition of the stack, the topmost data pair [4,1] in the current stack is popped.

[0091] The number of data pairs in the stack is counted as 0, therefore nodes OP4, OP5, ..., OP are determined. 15 A subset of level 1 nodes;

[0092] The traversal concludes, revealing that the operator subgroups in the neural network model comprise two level-1 operator subgroups and one level-2 operator subgroup. One level-1 operator subgroup includes operators op1, op2, and op3; the other level-1 operator subgroup includes operators op4, op5, op6, op7, op8, op9, op10, op11, and op15. The level-2 operator subgroup includes op12, op13, and op14.

[0093] S6. Input the target image into the neural network model and record the input and output data of the operator subgroup.

[0094] When performing image detection using a neural network model, the image's feature data is passed between operators. It is necessary to record the input and output data of the operator subgroups before proceeding to the next step of image processing and calculation.

[0095] In this embodiment, the input data of the operator subgroup is the output data of the starting operator corresponding to the starting node, and the output data of the operator subgroup is the input data of the ending operator corresponding to the ending node. The method provided in this application can record only the input and output data of the operator subgroups in the neural network model, without needing to record the input and output data of all operators in the neural network model. This significantly reduces the amount of data to be recorded.

[0096] S7. Based on the input data and output data, the target image is detected using the neural network model, and the detection result is output.

[0097] In this embodiment, when it is necessary to detect faces in the target image using the neural network model, features in the target image need to be extracted. These features mainly include color, texture, and edges. Therefore, the input and output data mainly include color data, texture data, and edge data. The detection of the target image is completed through the transmission of the input and output data between operator subgroups in the neural network model, and a face detection result is output.

[0098] It should be noted that if the modules / units integrated in the electronic device 5 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program code can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0099] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0100] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0102] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device in software or hardware. The terms "second class" are used to indicate names and do not indicate any particular order.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image detection method, applied in electronic devices, characterized in that, The image detection method includes: Obtain a neural network model, wherein the neural network model includes n operators; Based on the logical operation relationships between the n operators, the neural network model is mapped to a singly linked list structure, wherein the singly linked list structure includes n nodes, and the n nodes in the singly linked list are OP1, OP2, ..., OP1. i ,…,OP n ; Scan each node in the singly linked list structure and count the output and input of each node in the neural network model; Based on the output and input of each node, a data pair vector corresponding to the singly linked list structure is established, wherein the data pair vector is [[a1,b1],[a2,b2],…,[ai,bi],…,[an,bn]], and ai is the node OP. i The output quantity, a i Indicates the node OP i The corresponding operator is the number of operators that receive the data when it transmits the data outward in the neural network model; bi is the node OP. i The input quantity, bi, represents the node OP. i The number of other operators counted when the corresponding operator receives data transmitted by other operators in the neural network model; Based on the data pair vectors, determine the operator subgroups in the neural network model; Input the target image into the neural network model, and record the input and output data of the operator subgroup; Based on the input and output data, the target image is detected using the neural network model, and the detection results are output.

2. The image detection method according to claim 1, characterized in that, Determining the operator subgroups in the neural network model based on the data pair vectors includes: Multiple node subsets are determined based on the data pair vector; The operator subgroups in the neural network model are determined based on the plurality of node subsets.

3. The image detection method according to claim 2, characterized in that, Determining multiple node subsets based on the data pair vector includes: Step 51: Traverse the data pairs in the data pair vector starting from [a1, b1]; Step 52: Determine whether the data pair meets the push-on condition. If the data pair does not meet the push-on condition, proceed to step 54. Step 53: If the data pair meets the push-on conditions, push the data pair onto the stack; Step 54: Determine whether the data pair meets the pop stack condition. If the data pair does not meet the pop stack condition, proceed to step 57. Step 55: If the data pair satisfies the pop stack condition, pop the topmost data pair in the current stack and count the number m of the remaining data pairs in the stack; Step 56: Determine the starting node corresponding to the topmost data pair in the stack, and determine the ending node corresponding to the data pair that satisfies the pop stack condition. Set the subset consisting of all nodes from the starting node to the ending node as the m+1 level node subset. Step 57: Determine whether there are still untraversed data pairs in the data pair vector. If there are still untraversed data pairs in the data pair vector, return to step 51 and continue traversing the next data pair in the data pair vector. If all data pairs in the data pair vector have been traversed, end the traversal.

4. The image detection method according to claim 2, characterized in that, Determining the operator subgroup in the neural network model based on the plurality of node subsets includes: Traverse the multiple node subsets; If no node in any of the multiple node subsets is repeated from any node in any of the other node subsets, then that node subset is determined to be an operator subgroup in the neural network model.

5. The image detection method according to claim 2, characterized in that, Determining the operator subgroups in the neural network model based on the multiple node subsets further includes: By traversing the plurality of node subsets, a plurality of first node subsets and a plurality of second node subsets are obtained, wherein the second node subsets truly contain the first node subsets; Remove all nodes from the first node subset within the second node subset; The set of remaining nodes in the second node subset is defined as the operator subgroup of the neural network model.

6. The image detection method according to claim 5, characterized in that, Determining the operator subgroups in the neural network model based on the multiple node subsets further includes: The plurality of first node subsets are defined as operator subgroups of the neural network model.

7. The image detection method according to claim 3, characterized in that: If the data pair is [a] i ,b i b in ] i Greater than or equal to 2, determine the data pair [a] i ,b i The conditions for popping from the stack are met; If the data pair is [a] i ,b i a in ] i Greater than or equal to 2, determine the data pair [a] i ,b i The conditions for pushing onto the stack are met.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the image detection method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the image detection method as described in any one of claims 1 to 7.

Images