Method and device for locating target object in image

Abstract

An embodiment of the present application provides a method and device for locating a target object in an image, wherein the method includes: acquiring an image to be detected; obtaining a first processed image and a second processed image based on the image to be detected; superimposing the first processed image and the second processed image to obtain a superimposed result image; detecting the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected. Since the scheme first pre-processes the image to be detected to obtain a first processed image and a second processed image with richer image features, respectively; and then superimposing the above two images to combine the different image feature advantages of the two images, the feature information of the image input to the neural network model is further enhanced and enriched; and then the improved neural network model can more effectively utilize the above image feature information to determine the target positioning frame, thereby solving the technical problems of slow positioning speed and poor accuracy in the existing methods.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for locating a target object in an image. Background Art

[0002] When using a neural network to process image data, it is usually necessary to first find and determine a positioning frame, and use the positioning frame to locate the target object in the image (or the main content of the image) to determine and identify the position of the target object in the image, and then further process the image data at that position, for example, identify the image content at that position, or extract feature data of the image at that position, etc.

[0003] At present, in order to determine the target positioning frame used to identify the location area of ​​the target object in the image, most of the target detection models based on SSD (Single Shot MultiBox Detector) are used to detect and process the image to be detected, so as to identify and determine the target positioning frame in the image. However, the above target detection model is limited by the implementation mechanism, and its model structure is relatively complex. As a result, when the above target detection model is used for image positioning, the implementation efficiency is relatively low and the accuracy is relatively poor, which is not suitable for deployment in devices with low hardware conditions such as embedded mobile devices. In summary, when the existing methods for positioning target objects in images are implemented, there are often technical problems such as slow positioning speed and poor accuracy.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for locating a target object in an image to solve the technical problems of slow positioning speed and poor accuracy in existing methods, and achieve the technical effect of being suitable for deployment and application in operating environments such as embedded mobile devices, and efficiently and automatically identifying and determining the target positioning frame in the image.

[0006] The present application provides a method for locating a target object in an image, comprising:

[0007] Acquire the image to be detected;

[0008] Obtaining a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced boundaries, and the second processed image is an RGB image with enhanced contrast;

[0009] superimposing the first processed image and the second processed image to obtain a superimposed result image;

[0010] The superimposed result image is detected using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image region where a target object in the image to be detected is located.

[0011] In one embodiment, obtaining a first processed image according to the image to be detected includes:

[0012] Obtaining a grayscale image and a boundary image according to the image to be detected;

[0013] The grayscale image and the boundary image are added to obtain the first processed image.

[0014] In one embodiment, obtaining a second processed image according to the image to be detected includes:

[0015] Determine the image convolution kernel;

[0016] The image to be detected is subjected to convolution kernel filtering using the image convolution kernel to obtain the second processed image.

[0017] In one embodiment, superimposing the first processed image and the second processed image to obtain a superimposed result image includes:

[0018] The first processed image and the second processed image are convoluted and superimposed by using a preset connection method to obtain the superimposed result image.

[0019] In one implementation, the preset connection mode includes a route connection mode.

[0020] In one embodiment, the stacked result image is detected using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image area where a target object in the image to be detected is located, including:

[0021] According to the superimposed result image, a plurality of feature maps are generated using the improved neural network model, wherein the feature maps are used to detect the target object;

[0022] According to the feature map, generating multiple candidate boxes using the improved neural network model;

[0023] The target positioning frame is selected from the multiple candidate frames.

[0024] In one embodiment, the improved neural network model is a model based on the mobilenet v2 structure, and the number of feature maps generated and the number of candidate boxes generated are adjusted.

[0025] In one embodiment, the improved neural network model is obtained in the following manner:

[0026] The feature map in the neural network model is modified to adjust the number of multiple feature maps generated by the neural network model to 8.

[0027] In one embodiment, the improved neural network model is used to generate a plurality of feature maps, including:

[0028] Using the improved neural network model, eight convolution kernels are generated through back propagation training; wherein each convolution kernel corresponds to a dimensional feature;

[0029] Using the 8 convolution kernels, the corresponding 8 feature maps are obtained.

[0030] In one embodiment, the improved neural network model is obtained in the following manner:

[0031] Modify the setting parameters in the neural network model to adjust the number of multiple candidate boxes generated by the neural network model to 3.

[0032] In one embodiment, generating multiple candidate boxes using the improved neural network model includes:

[0033] Using the improved neural network model, three candidate boxes are obtained through regression processing; wherein the candidate boxes respectively carry position parameters and confidence probabilities of the candidate boxes, and the confidence probabilities are used to characterize the probability of the target object existing in the candidate boxes.

[0034] In one embodiment, selecting the target positioning frame from the multiple candidate frames includes:

[0035] From the three candidate frames, a candidate frame with the highest confidence probability is selected as the target positioning frame.

[0036] The present application also provides a device for locating a target object in an image, including:

[0037] An acquisition module, used for acquiring an image to be detected;

[0038] A pre-processing module, used for obtaining a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced boundary, and the second processed image is an RGB image with enhanced contrast;

[0039] A superposition module, used for superimposing the first processed image and the second processed image to obtain a superposition result image;

[0040] A determination module is used to detect the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image area where a target object in the image to be detected is located.

[0041] An embodiment of the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, realizes acquiring an image to be detected; obtaining a first processed image and a second processed image based on the image to be detected, wherein the first processed image is a grayscale image with boundary enhancement, and the second processed image is an RGB image with contrast enhancement; superimposing the first processed image and the second processed image to obtain a superimposed result image; and detecting the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located.

[0042] In an embodiment of the present application, by first pre-processing the image to be detected, a first processed image and a second processed image with richer image features and more dimensions are obtained respectively; then, by superimposing the above two images to comprehensively utilize the different image feature advantages of the two images, the feature information of the image input to the neural network model is further enhanced and enriched; and then, through the improved neural network model, the above image feature information can be more effectively utilized to determine the target positioning frame, thereby solving the technical problems of slow positioning speed and poor accuracy in the existing methods, and achieving the technical effect of being suitable for deployment and application in operating environments such as embedded mobile devices, and efficiently and automatically identifying and determining the target positioning frame in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 is a processing flow chart of a method for locating a target object in an image provided in an embodiment of the present application;

[0045] Figure 2 is a structural diagram of a device for locating a target object in an image provided according to an embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device based on the method for locating a target object in an image provided by an embodiment of the present application;

[0047] Figure 4 is a schematic diagram of three candidate frames obtained by applying the method and device for locating a target object in an image provided by an embodiment of the present application in a scene example;

[0048] Figure 5 It is a schematic diagram of a target positioning frame determined by applying the method and device for positioning a target object in an image provided by an embodiment of the present application in a scene example. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0050] Considering the existing image positioning methods, the images to be detected are often not pre-processed in advance, but the images to be detected are directly input as input data into the trained neural network model (such as the target detection model based on SSD) for detection and recognition, so as to determine the target positioning frame that can characterize the location information of the target object in the image to be detected. When implementing the above method, the feature information of the image to be detected is not considered to be strengthened in a targeted manner, but all depends on the neural network model to extract and process the feature information in the image to be detected, resulting in the structure of the trained and used neural network model being relatively complex, and the speed is relatively slow and the accuracy is relatively poor when the above neural network model is used for image positioning. And because the structure of the neural network model relied on is relatively complex, it is bound to occupy more computing resources during implementation, and the requirements for hardware equipment are relatively high, resulting in it being unsuitable for deployment and application in operating environments such as embedded mobile devices with relatively low hardware conditions. In summary, the existing methods often have technical problems of slow positioning speed and poor accuracy when they are specifically implemented.

[0051] In response to the root cause of the above-mentioned technical problems, the present application considers pre-processing the image to be detected first to enrich and strengthen the feature information in the image data input into the neural network model. Based on the fact that the image data input into the neural network model has richer and more obvious feature information, the structure and setting of the neural network model can be simplified and improved accordingly. Then, the improved neural network model can be used to more effectively utilize the enriched and enhanced image feature information to quickly and accurately determine the target positioning frame, thereby solving the technical problems of slow positioning speed and poor accuracy in the existing methods, and achieving the technical effect of being suitable for deployment and application in low-hardware operating environments such as embedded mobile devices, and efficiently and automatically identifying and determining the target positioning frame in the image.

[0052] Based on the above thinking, the present application embodiment provides a method for locating a target object in an image. Figure 1 The processing flow chart of the method for locating a target object in an image provided in accordance with an embodiment of the present application is shown. The method for locating a target object in an image provided in an embodiment of the present application may include the following steps during specific implementation.

[0053] S11: Acquire the image to be detected.

[0054] In this embodiment, the image to be detected may be a three-channel RGB image, wherein the image to be detected contains image content to be identified or image data of a target object.

[0055] The above-mentioned image content or target object to be identified can be understood as a content object that needs to be detected and identified from the image to be detected based on a specific application scenario and implementation requirements. For example, for face recognition, the image to be detected can be an image containing a face, and the corresponding target object can be the face in the image. For another example, for license plate recognition, the image to be detected can be an image containing a license plate, and the corresponding target object can be the license plate in the image. Of course, the images to be detected and the target objects to be detected and identified listed above are only for better illustrating the implementation methods of the present application. During specific implementation, other types of images can be selected as images to be detected, and other types of content objects can be selected as target objects according to specific circumstances and construction requirements. This application is not limited to this.

[0056] S12: Obtain a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced edges, and the second processed image is an RGB image with enhanced contrast.

[0057] In one embodiment, it is considered that the existing methods often do not pre-process the acquired image to be detected, but directly input it as input data into the neural network model to locate the image target. As a result, the neural network model needs to occupy more resource costs and time costs when extracting image features for subsequent analysis and processing, which increases the burden of the neural network model, resulting in the neural network model designed and used by the existing method being more complex and the positioning speed being relatively slow during operation. Based on the above situation, the present embodiment first pre-processes the image to be detected to obtain a first processed image and a second processed image with different image feature advantages, respectively, and then the first processed image and the second processed image can be comprehensively used to replace the unprocessed image to be detected to enrich and strengthen the image feature information subsequently input into the neural network model.

[0058] In this embodiment, considering that the grayscale image with enhanced boundaries usually has better boundary feature information than the unprocessed image, and this better boundary feature information is helpful for the subsequent detection and processing of the image by the neural network model, the detection accuracy of the neural network model can be improved, and the image features can be better extracted. Therefore, the grayscale image with enhanced boundaries can be used as the first processed image, so that more accurate image features can be extracted through the neural network model later.

[0059] In this embodiment, it is considered that the contrast-enhanced RGB image generally has a stronger color contrast than the unprocessed image, and the neural network model is more sensitive to color difference. For example, the color difference saturation information is referenced in the regression process of the neural network model. Therefore, the contrast-enhanced RGB image can be used as the second processed image, so that richer (more dimensional) and more accurate image features can be extracted through the neural network model later.

[0060] In one embodiment, obtaining a first processed image according to the image to be detected may include the following contents during specific implementation:

[0061] S1: Obtaining a grayscale image and a boundary image according to the image to be detected;

[0062] S2: Add the grayscale image and the boundary image to obtain the first processed image.

[0063] In one embodiment, during specific implementation, the grayscale processing of the above-mentioned image to be detected can be performed according to the following formula to obtain a grayscale image:

[0064] Gray=0.30*R+0.59*G+0.11*B

[0065] Among them, Gray can be specifically expressed as a grayscale value, R can be specifically expressed as a value of a red channel, G can be specifically expressed as a value of a green channel, and B can be specifically expressed as a value of a blue channel.

[0066] In one embodiment, during specific implementation, the boundary of the image to be detected may be enhanced according to the following formula to obtain a boundary image:

[0067]

[0068] Among them, G x Specifically, G can represent the boundary of the image to be detected on x. y Specifically, it can represent the boundary of the image to be detected on y.

[0069] In this embodiment, the grayscale image and the boundary image are added to obtain a grayscale image with enhanced boundary as the first processed image, thereby achieving the effect of enhancing the feature information of the image boundary.

[0070] In one embodiment, obtaining the second processed image according to the image to be detected may include the following steps during implementation:

[0071] S1: Determine the image convolution kernel;

[0072] S2: Using the image convolution kernel to perform convolution kernel filtering on the image to be detected to obtain the second processed image.

[0073] In one embodiment, in order to enhance the contrast of the image and make the color difference in the image more obvious, in a specific implementation, the above-mentioned image convolution kernel can be determined and expressed as follows:

[0074]

[0075] Of course, it should be noted that the above-listed image convolution kernels are only schematic illustrations. In specific implementation, other operators can be determined as the above-mentioned image convolution kernels according to specific circumstances and accuracy requirements. This application does not limit this.

[0076] In this embodiment, the above-mentioned image convolution kernel can be used to perform convolution kernel filtering processing on the image to be detected, enhance the contrast of the image, and obtain a contrast-enhanced RGB image as the second processed image, thereby enhancing the contrast of the image so that the subsequent neural network model can extract more dimensional image feature information.

[0077] S13: Superimposing the first processed image and the second processed image to obtain a superimposed result image.

[0078] In one embodiment, in order to comprehensively utilize the different advantages of the image features of the first processed image and the second processed image, the first processed image and the second processed image may be superimposed to obtain a superimposed result image. The superimposed result image has the image feature advantages of the first processed image based on image boundary enhancement, and also has the image feature advantages of the second processed image based on contrast enhancement. The superimposed result image may be used as input data for a subsequent neural network model, so that the neural network model may obtain richer and more accurate image features based on the superimposed result image to locate the image target.

[0079] In one embodiment, the superposition of the first processed image and the second processed image to obtain the superposition result image may include: using a preset connection method to perform image convolution superposition on the first processed image and the second processed image to obtain the superposition result image.

[0080] In one embodiment, further, in order to improve the robustness of the algorithm, during specific implementation, the above-mentioned preset connection method can specifically be a route connection method. Among them, the above-mentioned route is specifically connected to the neural network model of the back end, so that the connection between different channels can be increased without increasing the obvious computational complexity. The first processed image and the second processed image from different channels are convolved and superimposed to obtain the above-mentioned superposition result image through the route connection method. Of course, it should be noted that the route connection methods listed above are only for better explanation of the implementation methods of the present application. During specific implementation, other connection methods can also be selected as the above-mentioned preset connection methods according to specific scenarios and requirements. This application is not limited to this.

[0081] S14: Detecting the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image region where a target object in the image to be detected is located.

[0082] In this embodiment, after obtaining the overlay result image, the overlay result image can be used as input, and the overlay result image can be detected and processed using a neural network model to determine a target positioning frame for identifying the graph area where the target object in the image to be detected is located, so as to complete the image target positioning of the image to be detected.

[0083] In this embodiment, the above-mentioned determination of the target positioning frame of the image to be detected can be understood as determining the position parameters (including the distance difference from the center) and size parameters (including the length and width of the positioning frame) of the positioning frame used to identify (or select) the image area where the target object in the image to be detected is located, and then the position parameters and size parameters of the above-mentioned target positioning frame can be used for further image data processing, for example, image recognition, etc.

[0084] In one embodiment, by using the superimposed result image that has the advantages of the image features of the first processed image and the second processed image as the input data of the neural network model, the image data acquired by the neural network model already has richer and more enhanced image features before the detection and processing, thereby reducing the task burden of the neural network model. At this time, the neural network model can be improved accordingly to simplify the structure of the neural network model, improve the processing speed of the neural network model while ensuring accuracy, and reduce the time and resource occupation of the neural network model, so that the operating environment such as embedded mobile devices with poor hardware conditions can also better use the improved neural network model for image target positioning based on limited hardware resources.

[0085] In one embodiment, the improved neural network model is used to detect the superimposed result image to determine the target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located. When implemented specifically, the following contents may be included:

[0086] S1: generating a plurality of feature maps using the improved neural network model according to the superimposed result image, wherein the feature maps are used to detect the target object;

[0087] S2: Generate multiple candidate boxes using the improved neural network model according to the feature map;

[0088] S3: Filtering the target positioning frame from the multiple candidate frames.

[0089] In one embodiment, the improved neural network model can specifically be a model in which the improved neural network model is based on the mobilenet v2 structure, and the number of feature maps generated and the number of candidate boxes generated are adjusted.

[0090] In this embodiment, the mobilenet v2 is a basic network of a neural network, which is a network obtained by making corresponding modifications to mobilenet v1. Specifically, for example, the mobilenet v2 uses inverted residuals, a model similar to the residual network, to increase the dimension of data input, which is actually to solve the problem that the activation function cannot be activated due to the disappearance of features.

[0091] In this embodiment, it is also necessary to explain that in order to ensure that the subsequent use of the above-mentioned improved neural network model can take into account both data processing accuracy and data processing efficiency, that is, to ensure that relatively rich image feature information can be obtained while reducing the number of candidate boxes used to determine the target position, the number of feature maps generated by the neural network model based on the mobilenet v2 structure, the number of candidate boxes generated and other parameters are adjusted accordingly. Then, the model based on the mobilenet v2 structure, in which the number of feature maps generated and the number of candidate boxes generated are adjusted, is used as an improved neural network model. Of course, it should be explained that the improved neural network model listed above is only a schematic description and should not constitute an improper limitation on the present application.

[0092] In one embodiment, considering that the superimposed result image input to the neural network model has more dimensional, richer, and clearer image feature information, the difficulty of extracting image features by the subsequent neural network model is reduced. Based on the above situation, combined with the data characteristics of the superimposed result image, it is considered that the neural network model can be improved and adjusted accordingly, and then the improved neural network model can be used to more efficiently extract image feature information and improve positioning speed; at the same time, it is guaranteed to have a high degree of accuracy.

[0093] In one embodiment, during specific implementation, the neural network model can be improved in the following manner to obtain an improved neural network model: the feature map in the neural network model is modified to adjust the number of multiple feature maps generated by the neural network model to 8.

[0094] In one embodiment, based on the improved neural network model, a plurality of feature maps are generated using the improved neural network model. When implemented specifically, the following contents may be included:

[0095] S1: Using the improved neural network model, 8 convolution kernels are generated through back propagation training; wherein the convolution kernels correspond to one dimensional feature respectively;

[0096] S2: Using the 8 convolution kernels, obtain the corresponding 8 feature maps.

[0097] In this implementation, each of the above convolution kernels corresponds to a dimensional feature, and different convolution kernels correspond to features of different levels.

[0098] In this embodiment, the above 8 feature maps come from 8 different convolution kernels. Correspondingly, the 8 feature maps are feature maps of 8 different dimensions, corresponding to the target object positions obtained based on different dimensions. In subsequent processing, the image pyramid theory can be used to combine the target object positions obtained based on different dimensions to obtain a more accurate target positioning frame.

[0099] In this embodiment, it should be noted that most existing methods only obtain 6 feature maps. However, since the input dimension of the present invention is greater and the superimposed result image has richer image feature information, 8 feature maps can be obtained through an improved neural network model to improve the accuracy of determining the target positioning frame.

[0100] In one embodiment, during specific implementation, the neural network model can be improved in the following manner to obtain an improved neural network model: modify the setting parameters in the neural network model to adjust the number of multiple candidate boxes generated by the neural network model to 3.

[0101] In one embodiment, based on the above-mentioned improved neural network model, multiple candidate boxes are generated using the improved neural network model. During specific implementation, the following contents may be included: using the improved neural network model, three candidate boxes are obtained through regression processing; wherein the candidate boxes respectively carry position parameters and confidence probabilities of the candidate boxes, and the confidence probability is used to characterize the probability that a target object exists in the candidate box.

[0102] In this embodiment, the three candidate frames mentioned above may specifically be candidate frames (or detection frames) designed along three different directions based on the same center point.

[0103] In this embodiment, during the specific implementation, the obtained candidate box carries the position parameters of the candidate box, for example, the above-mentioned position parameters may also include: the offset distance from the center point in the x-axis direction and the offset distance from the center point in the y-axis direction. The above-mentioned candidate boxes may also carry their respective confidence probabilities, wherein the above-mentioned confidence probabilities are obtained through regression optimization of the neural network model to indicate the probability value of whether there is a target object in the candidate box. Generally, the higher the confidence probability of a candidate box, the greater the possibility of the corresponding existence of the target object. In addition, the above-mentioned candidate box may also carry the type parameter of the target object in the candidate box. For example, identification information that identifies the target object in the candidate box as a face, etc. Of course, it should be noted that the parameter information carried by the candidate boxes listed above is only a schematic description. It should not constitute an improper limitation on the present application.

[0104] In this embodiment, it should be noted that the existing methods do not utilize the position offset parameters in the target object detection, and most of them generate 5 candidate boxes to identify the target object. However, in the embodiment of the present application, considering that the position offset parameters in the target object detection can be used as auxiliary parameters, it is not necessary to use so many candidate boxes. In order to reduce the complexity of the neural network model detection, improve the processing speed, and also in order not to reduce the scope of detection, the number of generated candidate boxes is changed to 3. This can also be better suitable for deployment in operating environments such as embedded mobile devices.

[0105] In one embodiment, selecting the target positioning frame from the multiple candidate frames may include: selecting a candidate frame with the highest confidence probability from the three candidate frames as the target positioning frame. In this way, the candidate frame with the highest accuracy can be selected more accurately as the target positioning frame.

[0106] From the above description, it can be seen that the method for locating the target object in the image provided by the embodiment of the present application, by first pre-processing the image to be detected, respectively obtains the first processed image and the second processed image with richer image features; then by superimposing the above two images, the different image feature advantages of the two images are combined to further enhance and enrich the feature information of the image input to the neural network model; and then the improved neural network model can more effectively utilize the above image feature information to determine the target positioning frame, thereby solving the technical problems of slow positioning speed and poor accuracy in the existing method, and achieving the technical effect of being suitable for deployment and application in operating environments such as embedded mobile devices, and efficiently and automatically identifying and determining the target positioning frame in the image; the first processed image and the second processed image are convoluted and superimposed by using a route connection method to obtain a superimposed result image with a more blocked dimension and better image features, thereby increasing the connection between different image channels and improving the robustness of the algorithm; the neural network model is also improved, and 8 feature maps are obtained by using the improved neural network model to obtain image features of more dimensions, thereby further improving the accuracy of image target positioning; the neural network model is also improved, and 3 candidate frames are generated by using the improved neural network model to screen and obtain the target positioning frame, thereby further improving the processing speed.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides a device for locating a target object in an image, as described in the following embodiments. Since the principle of solving the problem by the device for locating a target object in an image is similar to that of the method for locating a target object in an image, the implementation of the device for locating a target object in an image can refer to the implementation of the method for locating a target object in an image, and the repeated parts will not be repeated. As used below, the term "unit" or "module" may be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable. Please refer to Figure 2 , is a composition structure diagram of a device for locating a target object in an image provided in an embodiment of the present application. The device may specifically include: an acquisition module 21, a pre-processing module 22, a superposition module 23 and a determination module 24. The structure is described in detail below.

[0108] The acquisition module 21 can be specifically used to acquire the image to be detected;

[0109] The pre-processing module 22 may be specifically configured to obtain a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced edges, and the second processed image is an RGB image with enhanced contrast;

[0110] The superposition module 23 may be specifically configured to superimpose the first processed image and the second processed image to obtain a superimposed result image;

[0111] The determination module 24 can be specifically used to detect the superimposed result image using an improved neural network model to determine the target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located.

[0112] In one embodiment, in order to obtain the first processed image according to the image to be detected, the pre-processing module 22 may specifically include the following structural units:

[0113] A first processing unit, which can be specifically used to perform image boundary enhancement processing on the grayscale image, so as to obtain the first processed image and obtain a grayscale image and a boundary image according to the image to be detected;

[0114] The adding unit may be specifically configured to add the grayscale image and the boundary image to obtain the first processed image.

[0115] In one embodiment, in order to obtain the second processed image according to the image to be detected, the pre-processing module 22 may further include the following structural units:

[0116] A first determining unit, which can be specifically used to determine an image convolution kernel;

[0117] The second processing unit can be specifically configured to perform convolution kernel filtering on the image to be detected using the image convolution kernel to obtain the second processed image.

[0118] In one embodiment, in order to superimpose the first processed image and the second processed image to obtain a superimposed result image, the superimposition module 23 may specifically include the following structural units:

[0119] The superposition unit may be specifically configured to perform image convolution superposition on the first processed image and the second processed image using a preset connection method to obtain the superposition result image.

[0120] In one embodiment, the preset connection mode may specifically include a route connection mode, etc. Of course, the connection modes listed above are only exemplary and should not constitute an improper limitation on the present application.

[0121] In one embodiment, in order to detect the superimposed result image using the improved neural network model to determine the target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located, the above-mentioned determination module 24 may specifically include the following structural units:

[0122] A first generating unit may be specifically configured to generate a plurality of feature maps according to the superimposed result image using the improved neural network model, wherein the feature maps are used to detect a target object;

[0123] A second generating unit may be specifically configured to generate a plurality of candidate boxes according to the feature map using the improved neural network model;

[0124] The screening unit may be specifically configured to screen out the target positioning frame from the multiple candidate frames.

[0125] In one embodiment, the improved neural network model may be a model based on the mobilenet v2 structure, and the number of feature maps generated and the number of candidate boxes generated are adjusted.

[0126] In this embodiment, it should be noted that in order to ensure that the subsequent use of the above-mentioned improved neural network model can take into account both data processing accuracy and data processing efficiency, that is, while ensuring that relatively rich image feature information can be obtained, the number of candidate boxes used to determine the target position is reduced, and the parameters such as the number of feature maps generated by the neural network model based on the mobilenet v2 structure and the number of candidate boxes generated are adjusted accordingly. Then, the model based on the mobilenet v2 structure, in which the number of feature maps generated and the number of candidate boxes generated are adjusted, is used as an improved neural network model. Of course, it should be noted that the improved neural network model listed above is only a schematic description and should not constitute an improper limitation on the present application.

[0127] In one embodiment, the above-mentioned device may also include an improvement module for making targeted adjustments and modifications to the neural network model to obtain an improved neural network model.

[0128] In one embodiment, when the above-mentioned improvement module is implemented, the neural network model can be improved according to the following procedure: the feature map in the neural network model is modified to adjust the number of multiple feature maps generated by the neural network model to 8.

[0129] In one embodiment, based on the above-mentioned improved neural network model, the determination module 24 can be specifically implemented according to the following procedure to generate multiple feature maps using the improved neural network model: using the improved neural network model, 8 convolution kernels are generated through back propagation training; wherein the convolution kernels correspond to one dimensional feature respectively; using the 8 convolution kernels, the corresponding 8 feature maps are obtained.

[0130] In one embodiment, when the above-mentioned improvement module is implemented, the neural network model can also be improved in another way according to the following procedure: modify the setting parameters in the neural network model to adjust the number of multiple candidate boxes generated by the neural network model to 3.

[0131] In one embodiment, based on the above-mentioned improved neural network model, the determination module 24 can be implemented in accordance with the following procedure to generate multiple candidate boxes using the improved neural network model: using the improved neural network model, three candidate boxes are obtained through regression processing; wherein, the candidate boxes respectively carry the position parameters and confidence probabilities of the candidate boxes, and the confidence probability is used to characterize the probability that the target object exists in the candidate box.

[0132] In one embodiment, when the determination module 24 is implemented, the candidate box with the highest confidence probability can be selected from the three candidate boxes as the target positioning box.

[0133] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] It should be noted that the systems, devices, modules or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, in this specification, the above devices are described in various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0135] In addition, in this specification, adjectives such as first and second may only be used to distinguish one element or action from another element or action, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be interpreted as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0136] From the above description, it can be seen that the device for locating a target object in an image provided by the embodiment of the present application first pre-processes the image to be detected through the pre-processing module to obtain a first processed image and a second processed image with richer image features respectively; then the two images are superimposed through the superposition module to combine the different image feature advantages of the two images, and further enhance and enrich the feature information of the image input into the neural network model; and then the determination module can more effectively utilize the above image feature information by using the improved neural network model to determine the target positioning frame, thereby solving the technical problems of slow positioning speed and poor accuracy in the existing methods, and achieving the goal of being suitable for deployment and application in operating environments such as embedded mobile devices, and being efficient. The technical effect of automatically identifying and determining the target positioning frame in the image is achieved; the superposition module adopts the route connection method to perform image convolution superposition on the first processed image and the second processed image to obtain a superposition result image with a smaller dimension and better image features, which increases the connection between different image channels and improves the robustness of the algorithm; the neural network model is improved by the improvement module, and 8 feature maps are obtained by the determination module using the improved neural network model to obtain image features of more dimensions, thereby further improving the accuracy of image target positioning; the neural network model is improved by the improvement module, and 3 candidate frames are generated by the determination module using the improved neural network model to screen and obtain the target positioning frame, thereby further improving the processing speed.

[0137] The present application also provides an electronic device. Figure 3 The electronic device composition structure diagram of the method for locating a target object in an image provided by an embodiment of the present application is shown, and the electronic device may specifically include an input device 31, a processor 32, and a memory 33. Among them, the input device 31 can be specifically used to input an image to be detected. The processor 32 can be specifically used to obtain a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with boundary enhancement, and the second processed image is an RGB image with contrast enhancement; superimpose the first processed image and the second processed image to obtain a superimposed result image; use an improved neural network model to detect the superimposed result image to determine the target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located. The memory 33 can be specifically used to store the image to be detected input through the input device 31, the intermediate data generated by the processor 32, and related program instructions.

[0138] In this embodiment, the input device may specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input the original data and the program for processing these numbers into the computer. The input device can also obtain and receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor. The memory may specifically be a memory device used to store information in modern information technology. The memory may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function without a physical form is also called a memory, such as a RAM, a FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0139] In this embodiment, the functions and effects specifically realized by the electronic device can be explained in comparison with other embodiments and will not be described in detail here.

[0140] In an embodiment of the present application, a computer storage medium based on a method for locating a target object in an image is also provided, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the following are achieved: acquiring an image to be detected; obtaining a first processed image and a second processed image based on the image to be detected, wherein the first processed image is a grayscale image with boundary enhancement, and the second processed image is an RGB image with contrast enhancement; superimposing the first processed image and the second processed image to obtain a superimposed result image; and detecting the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify the image area where the target object in the image to be detected is located.

[0141] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD) or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0142] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0143] In a specific implementation scenario example, the method and device for locating a target object in an image provided by the embodiments of the present application are applied to determine a target positioning frame. The specific implementation process can be found in the following content.

[0144] Step 1: In traditional image processing, convolutional neural networks generally directly input 3-channel RGB images (i.e., images to be detected). In this example, improvements have been made to obtain better results. One improvement is: first, a grayscale image is obtained through image pre-processing, and the corresponding image boundary enhancement is performed in the grayscale image (i.e., the first processed image is obtained). This can obtain better boundary information and have better performance in the subsequent convolutional neural network. The second improvement is: based on the fact that neural networks are also sensitive to color, the regression process will refer to information such as color difference and saturation. In this example, the color of the image is also enhanced (i.e., the second processed image is obtained). Figuratively speaking, this is to make the color more color-difference, which is conducive to the God General Network to extract better features.

[0145] In improvement 1: extracting the corresponding grayscale image and the corresponding boundary enhancement formula can be specifically expressed as follows:

[0146] Gray=0.30*R+0.59*G+0.11*B

[0147]

[0148] Then, the obtained grayscale image and the boundary image are added to obtain a grayscale image with enhanced boundaries (ie, the first processed image).

[0149] In Improvement 2: The color enhancement of the image specifically uses the following calculation method to enhance the contrast:

[0150] In this example, the image is actually subjected to a fixed image convolution operation, and its convolution kernel can be specifically expressed as follows:

[0151] 0 -1 0 -1 5 -1 0 -1 0

[0152] After filtering with the convolution kernel, the image has enhanced contrast and the color difference is obvious. Generally speaking, the image input of the neural network is generally in the pixel dimension of 224 or 229. After such image enhancement, the feature dimension available to the network model is significantly increased.

[0153] Step 2: In this process, mobilenetv2 is used as the basic network. However, due to the limitations of mobilenet itself, corresponding improvements are made in this example (to obtain an improved neural network model).

[0154] In this example, it should be noted that Mobilenet V2 has made corresponding improvements to Mobilenet V1. For example, it uses inverted residuals, a model similar to residual networks, to increase the dimension of data input. In fact, it is to solve the problem that the activation function cannot be activated during training due to feature disappearance. This article further enhances this.

[0155] In this example, the route connection method is used to perform image convolution and superposition on multiple channels of the image input source (i.e., to obtain a superimposed result image). The advantage of this is that it increases the connection between the channels of the image without greatly increasing the computational complexity, which is conducive to improving the robustness of the algorithm. In fact, the essence is: increase the image input and enrich the image input features.

[0156] Step 3: In the image detection task, 8 feature maps are obtained using the improved neural network model for detecting target content.

[0157] In this example, using 8 feature maps is similar to the theory of image pyramid. However, the 8-dimensional image comes from 8 different convolution kernels, where the convolution kernel uses adding and stride to obtain features at different levels. Among them, the obtained feature maps of multiple dimensions can obtain the corresponding target position from different size levels. This increases the accuracy of image processing.

[0158] Step 4: After the image has been detected in the third step, the image can obtain the probability confidence of whether it is a certain target at each feature map position, but there is no target location information. Since the image is the best result obtained after the regression problem, the neural network model is improved, and candidate target boxes (i.e. candidate boxes) are designed from three directions of the center point. Compared with the original VGG-SSD model, this makes the target box selection accurate and reduces the computational complexity by about half. The three candidate boxes generated above can be referred to Figure 4 The figure shows a schematic diagram of three candidate frames obtained by applying the method and device for locating a target object in an image provided by an embodiment of the present application in a scene example. Then, according to the confidence probability of each candidate frame, the candidate frame with the highest confidence probability can be selected from the above three candidate frames as the target positioning frame. Figure 5 The diagram shown is a schematic diagram of a target positioning frame determined by applying the method and apparatus for positioning a target object in an image provided by an embodiment of the present application in a scene example.

[0159] Comparing this example with the existing image localization methods, the following conclusions can be drawn: First, since the input of the image source does not use the original RGB three-channel image, the improvement uses a grayscale image superimposed on the boundary contour map. On this basis, the source image is also enhanced in color. In essence, this is a signal enhancement of the features of the boundary information and color information of the image, and such an operation will achieve better results in feature extraction. Secondly, in the processing flow in the middle of the model, the convolution operation in the convolutional neural network is decomposed by convolution. The decomposition operation is very conducive to richer feature extraction and reduces the complexity of the model's calculation parameters. Although Mobilenet's pointwise and depthwise have the function of reducing computing resource consumption, they also have the disadvantage of losing feature information. At the same time, this disadvantage easily causes the training to enter an infinite loop and the algorithm cannot converge. This paper improves the use of route connection to perform image convolution superposition on multiple channels of the image input source. The advantage of this is that it increases the connection between the channels of the image without greatly increasing the computational complexity, which is conducive to improving the robustness of the algorithm. Then, in the detection part, the original SSD network uses 5 detection boxes to detect the corresponding targets. In fact, in object detection, because of the position offset as an auxiliary parameter, so many detection boxes are not needed. In order to improve this shortcoming, the network layer is modified in this example, and 3 detection boxes are used. While maintaining the same detection range, the detection complexity is reduced by about half. Finally, in the feature map part of the neural network used by the existing method, this example adds eight feature maps in an improved way. The advantage is that more dimensional features can be obtained to obtain better detection effects for large and small objects.

[0160] Through the above-mentioned scenario examples, the method and device for positioning the target object in the image provided by the embodiment of the present application are verified. By first pre-processing the image to be detected, a first processed image and a second processed image with richer image features are obtained respectively; then, by superimposing the above two images to combine the different image feature advantages of the two images, the feature information of the image input into the neural network model is further enhanced and enriched; and then, through the improved neural network model, the above-mentioned image feature information can be more effectively utilized to determine the target positioning frame, which indeed solves the technical problems of slow positioning speed and poor accuracy in the existing methods, and achieves the technical effect of being suitable for deployment and application in operating environments such as embedded mobile devices, and efficiently and automatically identifying and determining the target positioning frame in the image.

[0161] Although different specific embodiments are mentioned in the content of this application, this application is not limited to the situations described in the industry standards or embodiments, etc. Some industry standards or slightly modified implementation plans based on the implementation of the custom methods or embodiments can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above embodiments. The embodiments of data acquisition, processing, output, judgment methods, etc. after these modifications or deformations can still fall within the scope of the optional implementation plans of this application.

[0162] Although the present application provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0163] The devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by dividing them into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules. The device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0164] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0165] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0166] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0167] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0168] Although the present application has been described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present application without departing from the spirit of the present application. It is intended that the attached embodiments include these modifications and changes without departing from the present application.

Claims

1. A method for locating a target object in an image, characterized in that: include: Acquire the image to be detected; Obtaining a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced boundaries, and the second processed image is an RGB image with enhanced contrast; superimposing the first processed image and the second processed image to obtain a superimposed result image; Using an improved neural network model to detect the superimposed result image to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image area where a target object in the image to be detected is located; The detecting the superimposed result image by using the improved neural network model to determine the target positioning frame of the image to be detected includes: According to the superimposed result image, a plurality of feature maps are generated using the improved neural network model, wherein the feature maps are used to detect the target object; According to the feature map, generating multiple candidate boxes using the improved neural network model; Filtering the target positioning frame from the multiple candidate frames; The improved neural network model is a model based on the mobilenet v2 structure, and the number of generated feature maps and the number of generated candidate boxes are adjusted; Obtaining a first processed image according to the image to be detected includes: Obtaining a grayscale image and a boundary image according to the image to be detected; Adding the grayscale image and the boundary image to obtain the first processed image; Obtaining a second processed image according to the image to be detected, comprising: Determine the image convolution kernel; Using the image convolution kernel to perform convolution kernel filtering on the image to be detected to obtain the second processed image; Superimposing the first processed image and the second processed image to obtain a superimposed result image includes: Using a preset connection method, performing image convolution superposition on the first processed image and the second processed image to obtain the superposition result image; The preset connection mode includes a route connection mode.

2. The method according to claim 1, characterized in that The improved neural network model is obtained in the following manner: The feature map in the neural network model is modified to adjust the number of the multiple feature maps generated by the neural network model to 8.

3. The method according to claim 2, characterized in that The improved neural network model is used to generate a plurality of feature maps, including: Using the improved neural network model, eight convolution kernels are generated through back propagation training; wherein each of the convolution kernels corresponds to a dimensional feature; Using the 8 convolution kernels, the corresponding 8 feature maps are obtained.

4. The method according to claim 1, characterized in that: The improved neural network model is obtained in the following manner: Modify the setting parameters in the neural network model to adjust the number of multiple candidate boxes generated by the neural network model to 3.

5. The method according to claim 4, characterized in that The improved neural network model is used to generate multiple candidate boxes, including: Using the improved neural network model, three candidate boxes are obtained through regression processing; wherein the candidate boxes respectively carry position parameters and confidence probabilities of the candidate boxes, and the confidence probabilities are used to characterize the probability of the target object existing in the candidate boxes.

6. The method according to claim 5, characterized in that Filtering the target positioning frame from the multiple candidate frames includes: From the three candidate frames, a candidate frame with the highest confidence probability is selected as the target positioning frame.

7. A device for locating a target object in an image, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A pre-processing module, used for obtaining a first processed image and a second processed image according to the image to be detected, wherein the first processed image is a grayscale image with enhanced boundary, and the second processed image is an RGB image with enhanced contrast; A superposition module, used for superimposing the first processed image and the second processed image to obtain a superposition result image; A determination module, used to detect the superimposed result image using an improved neural network model to determine a target positioning frame of the image to be detected, wherein the target positioning frame is used to identify an image area where a target object in the image to be detected is located; The detecting the superimposed result image by using the improved neural network model to determine the target positioning frame of the image to be detected includes: According to the superimposed result image, a plurality of feature maps are generated using the improved neural network model, wherein the feature maps are used to detect the target object; According to the feature map, generating multiple candidate boxes using the improved neural network model; Filtering the target positioning frame from the multiple candidate frames; The improved neural network model is a model based on the mobilenet v2 structure, and the number of generated feature maps and the number of generated candidate boxes are adjusted; Obtaining a first processed image according to the image to be detected includes: Obtaining a grayscale image and a boundary image according to the image to be detected; Adding the grayscale image and the boundary image to obtain the first processed image; Obtaining a second processed image according to the image to be detected, comprising: Determine the image convolution kernel; Using the image convolution kernel to perform convolution kernel filtering on the image to be detected to obtain the second processed image; Superimposing the first processed image and the second processed image to obtain a superimposed result image includes: Using a preset connection method, performing image convolution superposition on the first processed image and the second processed image to obtain the superposition result image; The preset connection mode includes a route connection mode.

8. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the steps of the method according to any one of claims 1 to 6 are implemented.

Images