An object recognition method, device and inspection equipment

Abstract

The application relates to the technical field of image recognition, in particular to an object recognition method and device and a checking device, which are used for solving the problems that the precision and accuracy of the recognition result obtained by a traditional image search mode are unstable, the method is as follows: images of multiple angles of a to-be-recognized object are acquired; then, for the image of any angle in the multiple angles, a to-be-processed image is obtained based on the image of the angle, and the obtained multiple to-be-processed images are synchronously input into a preset image search model, so as to obtain the recognition result of the to-be-recognized object, wherein the to-be-processed image is a front image of the to-be-recognized object at the corresponding angle; in this way, the images of multiple angles of the to-be-recognized object are comprehensively processed, the image search is performed by using the image information of the multiple angles of the to-be-recognized object, the precision and accuracy of the recognition result of the to-be-recognized object can be improved, and the reliable characteristic is met.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an object recognition method, apparatus and inspection equipment. Background Technology

[0002] In the field of image recognition, image search is an essential tool, based on relevant technologies. Specifically, it involves acquiring an image containing the object to be identified, and then using an image search method to retrieve matching results from a pre-defined database, such as product information or descriptions.

[0003] However, in practical applications, it has been found that the accuracy and precision of the recognition results of the objects to be identified obtained by the above image search method are unstable. Therefore, there is an urgent need for an object recognition method. Summary of the Invention

[0004] This application provides an object recognition method, apparatus, and inspection device to improve the accuracy and precision of recognition results obtained using image search methods.

[0005] The specific technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide an object identification method, including:

[0007] Acquire images of the object to be identified from multiple angles;

[0008] For an image at any of the plurality of angles, an image to be processed is obtained based on the image at any of the angles, wherein the image to be processed is an orthogonal image of the object to be identified at that any angle;

[0009] The obtained multiple images to be processed are simultaneously input into a preset image search model to obtain the recognition result of the object to be identified.

[0010] By employing the object recognition method described above in the embodiments of this application, multi-angle images of the object to be recognized are comprehensively processed, and image search is performed using the summarized image information of the object to be recognized from multiple angles. This can improve the accuracy and precision of the recognition results of the object to be recognized obtained by image search, and meet the reliability characteristics.

[0011] In one possible implementation, obtaining the image to be processed based on the image at any angle includes:

[0012] Target detection is performed on the image from any angle to obtain the position and angle information of the detection box corresponding to the object to be identified in the image from any angle.

[0013] Based on the location information, the image of the object to be identified is segmented from the image at any angle;

[0014] Based on the angle information, the image of the object to be identified is rotated to obtain the image to be processed.

[0015] In one possible implementation, the location information includes multiple coordinate information; the step of segmenting the image of the object to be identified from the image at any angle based on the location information includes:

[0016] Iterate through each first pixel within the image at any given angle;

[0017] For each first pixel traversed, the angle values ​​of each angle with the first pixel as the vertex are determined based on the position information and the coordinate information of the currently traversed first pixel. The coordinate information of other points associated with any of the angles is selected from the plurality of coordinate information according to a preset selection rule.

[0018] If the sum of the determined angle values ​​is less than a preset value, then the first pixel is marked.

[0019] After the traversal is completed, the image of the object to be identified is segmented from the image at any angle based on each of the marked first pixels.

[0020] The above method can remove images irrelevant to the object to be identified from the image at that angle, and retain the image containing the object to be identified at that angle to the maximum extent, thereby improving the accuracy and precision of subsequent recognition results.

[0021] In one possible implementation, rotating the image of the object to be identified based on the angle information to obtain the image to be processed includes:

[0022] Based on the angle information, the image of the object to be identified is rotated according to a preset rotation rule to obtain an intermediate image, wherein the rotation rule includes a rotation center and a rotation direction;

[0023] Based on the angle information and the coordinate information corresponding to the rotation center, the rotated coordinate information corresponding to each first pixel in the intermediate image is determined;

[0024] Based on the rotated coordinate information corresponding to each first pixel, a new image coordinate system is constructed, and based on the pixel value corresponding to each first pixel, the pixel value corresponding to each second pixel in the intermediate image in the image coordinate system is determined.

[0025] The image to be processed is determined based on the pixel value corresponding to each second pixel.

[0026] Since the model typically achieves a much higher recognition accuracy for upright images than for tilted images, the above method rotates the image of the object to be recognized to an upright image at that angle, thereby improving the recognition precision and accuracy of the subsequent image search model.

[0027] Furthermore, by constructing a new image coordinate system and determining the image to be processed based on the pixel values ​​corresponding to each second pixel, the image information of the object to be identified can be preserved to the maximum extent, thereby making the identification results of the object to be identified obtained by image search based on the image to be processed more accurate.

[0028] In one possible implementation, determining the pixel value corresponding to each second pixel in the intermediate image within the image coordinate system based on the pixel values ​​corresponding to each first pixel includes:

[0029] Traverse each second pixel in the intermediate image within the image coordinate system. For each second pixel traversed, query the pixel value corresponding to the coordinate information of the currently traversed second pixel from the rotated coordinate information corresponding to each first pixel.

[0030] If the query finds a match, the pixel value of the first pixel will be used as the pixel value of the second pixel.

[0031] If no match is found, multiple target pixels are selected from the first pixels based on the coordinate information of the second pixel and the rotated coordinate information of each first pixel. The pixel value of the second pixel is determined based on the pixel values ​​of the multiple target pixels, wherein the distance between the target pixel and the second pixel is less than a distance threshold.

[0032] The above method determines the image to be processed based on the pixel value corresponding to each second pixel, so as to preserve the image information of the object to be identified to the greatest extent.

[0033] In one possible implementation, determining the pixel value of the second pixel based on the pixel values ​​of the plurality of target pixels includes:

[0034] The average pixel value of the plurality of target pixels is used as the pixel value of the second pixel; or

[0035] The weighted sum of the pixel values ​​of the plurality of target pixels is used as the pixel value of the second pixel, wherein the smaller the distance between the target pixel and the second pixel, the larger the weight.

[0036] In one possible implementation, the step of synchronously inputting the obtained multiple images to be processed into a preset image search model includes:

[0037] Based on the largest image size among the multiple images to be processed, the multiple images to be processed are sized uniformly to obtain each image to be stitched together.

[0038] The images to be stitched together are stitched together sequentially according to a preset stitching order to obtain the target image;

[0039] Input the target image into the image search model;

[0040] During the size unification process, if the image size of any of the multiple images to be processed is smaller than the largest image size, then based on the largest image size, a region to be filled is determined in at least two surrounding areas of the image to be processed, and a preset pixel value is used to fill the region with pixels.

[0041] The above method unifies the images to be identified from multiple angles into the same image size, and then stitches the images to be stitched together. This can summarize the image information of the object to be identified from multiple angles, and then perform image search based on the summarized image information of the object to be identified from multiple angles. This can improve the model's recognition accuracy of the object to be identified, and thus make the recognition result of the object to be identified obtained by image search based on the image to be processed more accurate.

[0042] In one possible implementation, after inputting the target image into the image search model, the method further includes:

[0043] Based on the image search model, the target image is subjected to feature aggregation processing to obtain the comprehensive feature information corresponding to the target image;

[0044] Based on the comprehensive feature information, target feature information with a matching degree greater than the matching threshold is queried from a preset database;

[0045] The object description information corresponding to the target feature information is used as the recognition result of the object to be identified.

[0046] Secondly, embodiments of this application provide an object recognition device, comprising:

[0047] The acquisition module is used to acquire images of the object to be identified from multiple angles.

[0048] The segmentation module is used to obtain a processing image based on the image at any one of the plurality of angles, wherein the processing image is an orthogonal image of the object to be identified at any one angle;

[0049] The image search module is used to synchronously input multiple images to be processed into a preset image search model in order to obtain the recognition result of the object to be identified.

[0050] In one possible implementation, the segmentation module is specifically used for:

[0051] Target detection is performed on the image from any angle to obtain the position and angle information of the detection box corresponding to the object to be identified in the image from any angle.

[0052] Based on the location information, the image of the object to be identified is segmented from the image at any angle;

[0053] Based on the angle information, the image of the object to be identified is rotated to obtain the image to be processed.

[0054] In one possible implementation, the location information includes multiple coordinate information; the segmentation module is specifically used for:

[0055] Iterate through each first pixel within the image at any given angle;

[0056] For each first pixel traversed, the angle values ​​of each angle with the first pixel as the vertex are determined based on the position information and the coordinate information of the currently traversed first pixel. The coordinate information of other points associated with any of the angles is selected from the plurality of coordinate information according to a preset selection rule.

[0057] If the sum of the determined angle values ​​is less than a preset value, then the first pixel is marked.

[0058] After the traversal is completed, the image of the object to be identified is segmented from the image at any angle based on each of the marked first pixels.

[0059] In one possible implementation, the segmentation module is specifically used for:

[0060] Based on the angle information, the image of the object to be identified is rotated according to a preset rotation rule to obtain an intermediate image, wherein the rotation rule includes a rotation center and a rotation direction;

[0061] Based on the angle information and the coordinate information corresponding to the rotation center, the rotated coordinate information corresponding to each first pixel in the intermediate image is determined;

[0062] Based on the rotated coordinate information corresponding to each first pixel, a new image coordinate system is constructed, and based on the pixel value corresponding to each first pixel, the pixel value corresponding to each second pixel in the intermediate image in the image coordinate system is determined.

[0063] The image to be processed is determined based on the pixel value corresponding to each second pixel.

[0064] In one possible implementation, the segmentation module is specifically used for:

[0065] Traverse each second pixel in the intermediate image within the image coordinate system. For each second pixel traversed, query the pixel value corresponding to the coordinate information of the currently traversed second pixel from the rotated coordinate information corresponding to each first pixel.

[0066] If the query finds a match, the pixel value of the first pixel will be used as the pixel value of the second pixel.

[0067] If no match is found, multiple target pixels are selected from the first pixels based on the coordinate information of the second pixel and the rotated coordinate information of each first pixel. The pixel value of the second pixel is determined based on the pixel values ​​of the multiple target pixels, wherein the distance between the target pixel and the second pixel is less than a distance threshold.

[0068] In one possible implementation, the segmentation module is specifically used for:

[0069] The average pixel value of the plurality of target pixels is used as the pixel value of the second pixel; or

[0070] The weighted sum of the pixel values ​​of the plurality of target pixels is used as the pixel value of the second pixel, wherein the smaller the distance between the target pixel and the second pixel, the larger the weight.

[0071] In one possible implementation, the graph search module is specifically used for:

[0072] Based on the largest image size among the multiple images to be processed, the multiple images to be processed are sized uniformly to obtain each image to be stitched together.

[0073] The images to be stitched together are stitched together sequentially according to a preset stitching order to obtain the target image;

[0074] Input the target image into the image search model;

[0075] During the size unification process, if the image size of any of the multiple images to be processed is smaller than the largest image size, then based on the largest image size, a region to be filled is determined in at least two surrounding areas of the image to be processed, and a preset pixel value is used to fill the region with pixels.

[0076] In one possible implementation, after inputting the target image into the image search model, the image search module is further configured to:

[0077] Based on the image search model, the target image is subjected to feature aggregation processing to obtain the comprehensive feature information corresponding to the target image;

[0078] Based on the comprehensive feature information, target feature information with a matching degree greater than the matching threshold is queried from a preset database;

[0079] The object description information corresponding to the target feature information is used as the recognition result of the object to be identified.

[0080] Thirdly, embodiments of this application provide an electronic device, including:

[0081] Image acquisition device, used to acquire images;

[0082] Memory is used to store computer programs or instructions;

[0083] A processor for executing a computer program or instructions in the memory, such that the method described in any of the first aspects above is performed.

[0084] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.

[0085] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of an application scenario in the embodiments of this application;

[0087] Figure 2 This is a flowchart illustrating an object recognition method according to an embodiment of this application;

[0088] Figure 3 This is a schematic diagram of a process for obtaining an image to be processed according to an embodiment of this application;

[0089] Figure 4 This is a schematic diagram showing the position and angle information of a detection frame in an embodiment of this application;

[0090] Figure 5 This is a schematic diagram of an image segmentation process in an embodiment of this application;

[0091] Figure 6This is a schematic diagram illustrating the processing principle of image segmentation in an embodiment of this application;

[0092] Figure 7 This is a schematic diagram illustrating a specific process for obtaining an image to be processed in an embodiment of this application;

[0093] Figure 8 This is a schematic diagram illustrating an image rotation process in an embodiment of this application;

[0094] Figure 9 This is a schematic diagram illustrating a process for determining the pixel values ​​of each pixel in an intermediate image according to an embodiment of this application.

[0095] Figure 10 This is a flowchart illustrating a multi-image synchronous input image search model in an embodiment of this application;

[0096] Figure 11 This is a schematic diagram of the network architecture of a CNN neural network model in an embodiment of this application;

[0097] Figure 12 This is a schematic diagram of a process for obtaining the recognition result of an object to be identified in an embodiment of this application;

[0098] Figure 13 This is a schematic diagram of the logical architecture of an object recognition device according to an embodiment of this application;

[0099] Figure 14 This is a schematic diagram of the physical architecture of the inspection device in the embodiments of this application. Detailed Implementation

[0100] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0101] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0102] The design concept of the embodiments of this application will be briefly introduced below.

[0103] This application relates to the field of image recognition technology, and mainly to an object recognition method, apparatus and inspection equipment.

[0104] In real-world scenarios, such as customs security checks and station entry security checks, it is often necessary to conduct security checks on passengers' personal belongings. With the help of relevant technologies, image search is an essential tool in the field of image recognition.

[0105] In the above scenarios, inspection equipment is used to capture images of each object to be identified within the field of view. Then, based on the acquired images, an image search method is used to retrieve the matching identification results, such as product information, from a pre-set database to complete the security check.

[0106] However, in practical applications, image search methods often use single-sided images of the object. Since a single-sided image contains limited information, the accuracy and precision of the recognition results obtained using this image search method are unstable.

[0107] In view of this, in order to solve the above problems, this application provides an object recognition method. In this application embodiment, images of the object to be recognized from multiple angles are acquired respectively; then, for the image of any one of the multiple angles, a processing image is obtained based on the image of that angle; the multiple processing images are simultaneously input into a preset image search model to obtain the recognition result of the object to be recognized, wherein the processing image is the orthogonal image of the object to be recognized at the corresponding angle; in this way, by comprehensively processing the images of the object to be recognized from multiple angles and using the image search information of the object to be recognized from multiple angles, the accuracy and precision of the recognition result of the object to be recognized obtained by image search can be improved, which meets the reliability characteristics.

[0108] It should be noted that the object to be identified is not limited to commodities, and the identification result is not limited to commodity information. The object to be identified can also be any form of item, such as animals or plants. Accordingly, the identification result can be the description information of the animal or plant, etc. This application does not make any specific limitations.

[0109] The following describes an application scenario of an optional object recognition method provided by an embodiment of this application, with reference to the accompanying drawings.

[0110] like Figure 1 As shown, this application scenario includes inspection equipment 10. The inspection equipment 10 includes a table and an image acquisition device (…). Figure 1 The locations shown are merely examples and are not specifically limited in this application. The tabletop is used to place the object to be identified; the image acquisition device is installed on the tabletop to acquire images within the field of view on the tabletop.

[0111] For example, in a customs security inspection scenario, a passenger places their personal belongings (referred to as the object to be identified in this application embodiment) on the table of the inspection device 10, and the inspection device 10 executes an object recognition method according to this application embodiment to obtain the recognition result of the object to be identified.

[0112] In some preferred embodiments, the platform must ensure that the portion falling within the field of view has a solid color background, such as black or white, to avoid the shooting background affecting subsequent item detection.

[0113] In some feasible embodiments, the image acquisition device described above may be a camera, video camera, or other electronic devices with photographic or video recording functions, such as a tablet computer or smartphone.

[0114] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible application scenarios. For example, the above application scenarios also include a server connected to the inspection device 10 via wired or wireless connection. The inspection device 10 sends the collected images containing each object to be identified to the server, and the server executes an object recognition method according to an embodiment of this application to obtain the recognition result corresponding to each object to be identified, and sends each recognition result to the inspection device 10. The inspection device 10 performs corresponding operations based on each recognition result, such as displaying each recognition result. For example, any application scenario that requires image recognition. It is understood that the embodiments of this application do not specifically limit the application scenarios.

[0115] In some feasible embodiments, the inspection device 10 described above can also be any form of electronic device with an image acquisition device; for example, the electronic device can be a smart terminal, smart mobile terminal, tablet computer, laptop computer, smart handheld device, personal computer (PC), computer, smart screen, display device, various wearable devices, personal digital assistant (PDA), etc.

[0116] After introducing the application scenarios of the embodiments of this application, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0117] See Figure 2 As shown in the figure, this application provides an object recognition method, the specific process of which is as follows:

[0118] Step 200: Acquire images of the object to be identified from multiple angles.

[0119] In specific implementation, during step 200, through Figure 1 The inspection device shown acquires images of the object to be identified from multiple angles. It should be noted that each image may contain not only one object to be identified, but also multiple objects. Subsequent embodiments of this application will only describe the object identification method in detail using a single object as an example; the processing methods for other objects to be identified are the same or similar.

[0120] Step 210: For an image at any of the multiple angles, obtain an image to be processed based on the image at that angle, wherein the image to be processed is an orthogonal image of the object to be identified at that angle.

[0121] In the embodiments of this application, see the following: Figure 3 As shown, when performing step 210, for an image at any of the multiple angles, the following steps are specifically performed to obtain the image to be processed corresponding to that angle, wherein the image to be processed is the orthogonal image of the object to be identified at that angle:

[0122] Step 2101: Perform target detection on the image at this angle to obtain the position and angle information of the detection box corresponding to the object to be identified in the image at this angle.

[0123] In practice, since the placement of the objects to be identified is often quite random, when performing step 2101, target detection is performed on the image at this angle to obtain the position and angle information of the detection box that selects the object to be identified. The angle information represents the tilt angle of the detection box.

[0124] For example, such as Figure 4 As shown, an image of the object A to be identified from any angle is used as an example.

[0125] In this embodiment of the application, when performing step 2101, target detection is performed on the image at this angle. The object to be identified in the image can be selected by using a rotating rectangle A (i.e., the aforementioned detection box), and the position information and angle information of the rotating rectangle A are obtained. The position information includes the coordinate information of the four corners of the rotating rectangle A (a, b, c, d in sequence); the angle information includes the tilt angle of the rotating rectangle A, that is, the angle required to rotate the image of the object to be identified into an upright image, denoted as θ.

[0126] Step 2102: Based on the location information, segment the image of the object to be identified from the image at that angle.

[0127] In this embodiment of the application, the aforementioned location information includes multiple coordinate information. Therefore, refer to... Figure 5As shown, in step 2102, the following steps are specifically performed: determining which first pixels in the image at that angle are located within the detection box, thereby obtaining the pixels in the image at that angle that belong to the object to be identified:

[0128] Step 400: Iterate through each first pixel within the image at that angle.

[0129] Step 410: For each first pixel traversed, the angle values ​​of each angle with the first pixel as the vertex are determined based on the position information and the coordinate information of the first pixel traversed. The coordinate information of other points associated with any angle is selected from multiple coordinate information according to a preset selection rule.

[0130] For example, see Figure 6 As shown, we will still take an image of the object A to be identified from any angle as an example.

[0131] Assume that the location information of the detection box corresponding to the object to be identified in the image at this angle includes four coordinates, which are denoted as a(x1, y1), b(x2, y2), c(x3, y3), and d(x4, y4) respectively.

[0132] In practice, each first pixel within the image at that angle is traversed.

[0133] Assume that there are 50 first pixels in the image at this angle, and the first pixel currently traversed is denoted as first pixel A, and the coordinate information of first pixel A is denoted as e(x, y).

[0134] Then, according to the selection rules, select ab, bc, cd, ca in sequence, and calculate the included angle values ​​of each angle (i.e., ∠aeb, ∠bec, ∠ced, ∠dea) with the first pixel A as the vertex.

[0135] In this embodiment of the application, the included angle ∠aeb can be represented by the following formula:

[0136]

[0137] Correspondingly, the other three included angles ∠bec, ∠ced, and ∠dea can be calculated by replacing the coordinates of the corresponding points with the above formula, and obtaining the included angle values ​​of ∠bec, ∠ced, and ∠dea.

[0138] Step 420: If the sum of the determined angle values ​​is less than a preset value, then mark the first pixel.

[0139] In this embodiment of the application, after obtaining the included angle value of each included angle through step 410, the sum of the included angle values ​​of the above included angles is determined. Then, the sum of the included angle values ​​is compared with a preset value, wherein the preset value can be 365, and 5 is the deviation value.

[0140] In one possible embodiment, if the sum of the determined included angle values ​​is less than a preset value, such as... Figure 6 As shown, if the first pixel A is located within the detection box that selects the object to be identified, then step 420 is executed to mark the first pixel A; then, the next first pixel is traversed.

[0141] In another possible embodiment, if the sum of the determined angle values ​​is greater than or equal to a preset value, it indicates that the first pixel A is outside the detection box that selects the object to be identified, and then the next first pixel is traversed.

[0142] In this way, each first pixel located within the detection box that selects the object to be identified can be filtered out from the image at that angle.

[0143] Step 430: After traversal is complete, based on each first pixel marked, segment the image of the object to be identified from the image at that angle.

[0144] In this embodiment of the application, after traversing all the first pixels in the image at that angle, step 430 is executed to segment each marked first pixel from the image at that angle, thereby obtaining the image of the object to be identified.

[0145] In this way, irrelevant images of the object to be identified can be removed from the image at that angle, while retaining the image at that angle that contains the object to be identified to the maximum extent, thereby improving the accuracy and precision of subsequent recognition results.

[0146] Step 2103: Based on the angle information, rotate the image of the object to be identified to obtain the image to be processed.

[0147] In the embodiments of this application, see the following: Figure 7 As shown, when executing step 2103, the following steps are specifically performed:

[0148] Step 500: Based on the angle information, rotate the image of the object to be identified according to the preset rotation rules to obtain an intermediate image, wherein the rotation rules include the rotation center and the rotation direction.

[0149] For example, see Figure 8 As shown, taking an image of the object to be identified at any angle as an example. In this embodiment, the rotation center can be selected as point a, and the rotation direction is counterclockwise; then, when executing step 500, the image of the object to be identified is rotated counterclockwise by θ (i.e., angle information) with the rotation center (point a) as the center, thereby rotating the object to be identified to the correct position and obtaining the intermediate image.

[0150] In practical applications, it has been found that machine recognition is usually highly accurate for upright images. However, when the image of the object to be recognized is deflected, the recognition accuracy of machine recognition is greatly reduced. Improving the recognition accuracy of machine learning through extensive training is difficult and labor-intensive. Therefore, in this embodiment, the workload is greatly reduced by executing step 500, and the processing method is simple and convenient.

[0151] Step 510: Based on the angle information and the coordinate information corresponding to the rotation center, determine the rotated coordinate information of each first pixel in the intermediate image.

[0152] In this embodiment of the application, since the image of the object to be identified has been rotated, the coordinates of the first pixel in the image of the object to be identified will also change. Therefore, step 510 is executed to determine the rotated coordinate information of each first pixel in the intermediate image.

[0153] In this embodiment, the first pixel A is still taken as an example; assuming that the first pixel A is any first pixel in the intermediate image, the rotated coordinate information (x', y') of the first pixel A can be represented by the following formulas:

[0154] x'=(x﹣x1)cosθ+(y﹣y1)sinθ

[0155] y'=(y﹣y1)cosθ﹣(x﹣x1)sinθ

[0156] Step 520: Based on the rotated coordinate information corresponding to each first pixel, construct a new image coordinate system, and based on the pixel value corresponding to each first pixel, determine the pixel value corresponding to each second pixel in the intermediate image within the image coordinate system.

[0157] In this embodiment, since the coordinates of pixels in an image are usually integers, and after performing the rotation operation in step 500, the coordinates of some first pixels in the intermediate image may be non-integer. Therefore, step 520 is performed to construct a new image coordinate system based on the rotated coordinate information corresponding to each first pixel, and to determine the pixel value of each second pixel in the intermediate image in the new image coordinate system based on the pixel value corresponding to each first pixel. This facilitates the subsequent summarization of multi-angle images of the object to be identified, enabling comprehensive processing of image information of the object to be identified from multiple angles, and preparing for improving the accuracy and precision of the image search model.

[0158] In specific implementation, when constructing a new image coordinate system based on the rotated coordinate information corresponding to each first pixel in step 520, the maximum and minimum values ​​of the horizontal coordinate can be selected from the rotated coordinate information corresponding to each first pixel and denoted as xmax and xmin, respectively, and the maximum and minimum values ​​of the vertical coordinate can be denoted as ymax and ymin, respectively. The selected xmax, xmin, ymax and ymin are then rounded. Then, xmin' = 0, xmax' = xmax - xmin, ymin' = 0, ymax' = ymax - ymin, that is, the upper left corner of the intermediate image is taken as the origin of the new image coordinate system.

[0159] Then, refer to Figure 9 As shown, when performing step 520, which involves determining the pixel value corresponding to each second pixel in the intermediate image based on the pixel value corresponding to each first pixel, the following steps are specifically performed:

[0160] Step 5201: Traverse each second pixel in the intermediate image within the image coordinate system. For each second pixel traversed, query the pixel value corresponding to the coordinate information of the currently traversed second pixel from the rotated coordinate information corresponding to each first pixel.

[0161] Step 5202: If found, the pixel value of the first pixel is determined as the pixel value of the second pixel.

[0162] Step 5203: If no target pixel is found, multiple target pixels are selected from each first pixel based on the coordinate information of the second pixel and the rotated coordinate information of each first pixel. The pixel value of the second pixel is determined based on the pixel values ​​of the multiple target pixels, wherein the distance between the target pixel and the second pixel is less than the distance threshold.

[0163] In specific implementation, when determining the pixel value of the second pixel based on the pixel values ​​of multiple target pixels in step 5203, the pixel value of the second pixel can be determined in any of the following ways:

[0164] Method 1: Take the average pixel value of multiple target pixels as the pixel value of the second pixel.

[0165] Method 2 involves using the weighted sum of the pixel values ​​of multiple target pixels as the pixel value of the second pixel, where the smaller the distance between the target pixel and the second pixel, the larger the weight.

[0166] For example, let's take an image of the object A to be identified from any angle as an example.

[0167] During steps 5201 to 5203, the pixel value corresponding to each second pixel in the intermediate image within the image coordinate system can be obtained through the following traversal processing logic:

[0168] Let i iterate between 0 and xmax':

[0169] Let j traverse between 0 and ymax':

[0170] Calculate the distance from each first pixel to the currently traversed second pixel (i, j), and select n target pixels whose distances are less than the distance threshold from the obtained distances;

[0171] Calculate the distance from n target pixels to the second pixel (i, j);

[0172] Calculate the RGB of the second pixel (i, j).

[0173] It is important to note that if there exists a first pixel whose distance to the currently traversed second pixel (i, j) is 0, then the RGB of the first pixel is determined as the RGB of the second pixel (i, j).

[0174] In the example above, assume n = 3, the RGB values ​​of the three target pixels are denoted as r1, r2, r3; g1, g2, g3; b1, b2, b3; and the distances between the three target pixels and the second pixel (i, j) are denoted as L1, L2, L3.

[0175] Therefore, in this embodiment of the application, when calculating the RGB of the second pixel (i, j) using method two, the RGB of the second pixel (i, j) can be represented by the following formula:

[0176] r=((l / L1)r1+(1 / L2)r2+(1 / L3)r3) / (1 / L1+1 / L2+1 / L3);

[0177] g=((l / L1)g1+(1 / L2)g2+(1 / L3)g3) / (1 / L1+1 / L2+1 / L3);

[0178] b=((l / L1)b1+(1 / L2)b2+(1 / L3)b3) / (1 / L1+1 / L2+1 / L3).

[0179] Where (l / L1) / (1 / L1+1 / L2+1 / L3) is the weight of r1, and the rest are similar and will not be repeated.

[0180] Step 530: Determine the image to be processed based on the pixel value corresponding to each second pixel.

[0181] Thus, by performing the above steps, the image to be processed corresponding to that angle can be obtained. This image to be processed is the orthogonal image of the object to be identified at that angle. Correspondingly, by performing the above steps on images from multiple angles, images to be processed corresponding to multiple angles can be obtained. It should be noted that the image to be processed for each angle includes the orthogonal image of the object to be identified at that angle. Inputting the orthogonal images of the object to be identified from multiple angles into the image search model in subsequent step 220 can yield a more accurate identification result for the object to be identified.

[0182] Step 220: Simultaneously input the obtained multiple images to be processed into the preset image search model to obtain the recognition result of the object to be identified.

[0183] In the embodiments of this application, see the following: Figure 10 As shown, when executing step 220, the following steps are specifically performed:

[0184] Step 2201: Based on the largest image size among the multiple images to be processed, the sizes of the multiple images to be processed are unified to obtain each image to be stitched; wherein, during the size unification process, if the image size of any of the multiple images to be processed is smaller than the largest image size, then based on the largest image size, a region to be filled is determined in at least two surrounding areas of any image to be processed, and a preset pixel value is used to fill the region with pixels.

[0185] In this embodiment of the application, when performing step 2201, if it is determined that the image size of a certain image to be processed is smaller than the maximum image size, then by using pixel padding, such as using pixels with zero pixel values, the image size of the insufficient image to be processed is filled to the maximum image size, ensuring that the object to be identified is not distorted, thereby improving the accuracy and precision of subsequent image search.

[0186] Step 2202: According to the preset splicing order, the images to be spliced ​​are spliced ​​sequentially to obtain the target image.

[0187] In this embodiment of the application, the image to be processed is an image of both the front and back sides of the object to be identified. If the front image is represented by 224×224×3 and the back image is represented by 224×224×3, then the target image obtained after performing step 2202 can be represented by a three-dimensional tensor of 224×224×6.

[0188] Step 2203: Input the target image into the image search model described above.

[0189] In this embodiment, the image search model described above can be constructed using any type of neural network, and no specific limitation is made here. For example, see [link to relevant documentation]. Figure 11 As shown, taking the image search model as a Convolutional Neural Network (CNN) model as an example, this CNN neural network model can be trained using a triplet loss function, so that the model has high similarity for feature vectors of the same class and low similarity for feature vectors of different classes. The specific training process can adopt a general training method, which will not be elaborated here.

[0190] Thus, in the actual use phase of the model, after executing step 2203 and inputting the target image into the aforementioned image search model, refer to... Figure 12 As shown, the recognition result of the object to be identified is obtained by performing the following steps:

[0191] Step 800: Based on the image search model, perform feature aggregation processing on the target image to obtain the comprehensive feature information corresponding to the target image.

[0192] Step 810: Based on the comprehensive feature information, query the target feature information from the preset database that has a matching degree greater than the matching threshold.

[0193] In practice, when executing step 810, the feature information in the database that has the highest matching degree with the comprehensive feature information is usually output as the target feature information.

[0194] Step 820: Use the object description information corresponding to the target feature information as the recognition result of the object to be identified.

[0195] For example, let's take the case where the image to be processed is the front and back images of the object to be identified.

[0196] Assuming the image search model is a CNN neural network model, then in step 2203, the three-dimensional tensor corresponding to the stitched target image is input into the CNN neural network model.

[0197] Then, step 800 is executed, the CNN neural network model outputs a one-dimensional feature vector, namely the aforementioned comprehensive feature information; and step 810 is executed, the one-dimensional feature vector is compared with each feature vector in the database, so as to find the feature vector with the highest similarity to the one-dimensional feature vector.

[0198] Finally, step 820 is executed to output the object description information corresponding to the feature vector with the highest similarity, thus obtaining the recognition result of the object to be identified. The object description information may include the standard image, introduction information, prompt information, logistics information, etc. of the object to be identified. If the object to be identified is a product, the object description information may also include brand information, product batch number information, logistics traceability information, etc.

[0199] Based on the same inventive concept, see [reference] Figure 13 As shown, this application provides an object recognition device, including:

[0200] The acquisition module 1310 is used to acquire images of the object to be identified from multiple angles.

[0201] The segmentation module 1320 is used to obtain a to-be-processed image based on an image at any of the multiple angles, wherein the to-be-processed image is an orthogonal image of the object to be identified at any of the angles.

[0202] Image search module 1330 is used to synchronously input multiple images to be processed into a preset image search model in order to obtain the recognition result of the object to be identified.

[0203] In one possible implementation, the segmentation module 1320 is specifically used for:

[0204] Target detection is performed on the image from any angle to obtain the position and angle information of the detection box corresponding to the object to be identified in the image from any angle.

[0205] Based on the location information, the image of the object to be identified is segmented from the image at any angle;

[0206] Based on the angle information, the image of the object to be identified is rotated to obtain the image to be processed.

[0207] In one possible implementation, the location information includes multiple coordinate information; the segmentation module 1320 is specifically used for:

[0208] Iterate through each first pixel within the image at any given angle;

[0209] For each first pixel traversed, the angle values ​​of each angle with the first pixel as the vertex are determined based on the position information and the coordinate information of the currently traversed first pixel. The coordinate information of other points associated with any of the angles is selected from the plurality of coordinate information according to a preset selection rule.

[0210] If the sum of the determined angle values ​​is less than a preset value, then the first pixel is marked.

[0211] After the traversal is completed, the image of the object to be identified is segmented from the image at any angle based on each of the marked first pixels.

[0212] In one possible implementation, the segmentation module 1320 is specifically used for:

[0213] Based on the angle information, the image of the object to be identified is rotated according to a preset rotation rule to obtain an intermediate image, wherein the rotation rule includes a rotation center and a rotation direction;

[0214] Based on the angle information and the coordinate information corresponding to the rotation center, the rotated coordinate information corresponding to each first pixel in the intermediate image is determined;

[0215] Based on the rotated coordinate information corresponding to each first pixel, a new image coordinate system is constructed, and based on the pixel value corresponding to each first pixel, the pixel value corresponding to each second pixel in the intermediate image in the image coordinate system is determined.

[0216] The image to be processed is determined based on the pixel value corresponding to each second pixel.

[0217] In one possible implementation, the segmentation module 1320 is specifically used for:

[0218] Traverse each second pixel in the intermediate image within the image coordinate system. For each second pixel traversed, query the pixel value corresponding to the coordinate information of the currently traversed second pixel from the rotated coordinate information corresponding to each first pixel.

[0219] If the query finds a match, the pixel value of the first pixel will be used as the pixel value of the second pixel.

[0220] If no match is found, multiple target pixels are selected from the first pixels based on the coordinate information of the second pixel and the rotated coordinate information of each first pixel. The pixel value of the second pixel is determined based on the pixel values ​​of the multiple target pixels, wherein the distance between the target pixel and the second pixel is less than a distance threshold.

[0221] In one possible implementation, the segmentation module 1320 is specifically used for:

[0222] The average pixel value of the plurality of target pixels is used as the pixel value of the second pixel; or

[0223] The weighted sum of the pixel values ​​of the plurality of target pixels is used as the pixel value of the second pixel, wherein the smaller the distance between the target pixel and the second pixel, the larger the weight.

[0224] In one possible implementation, the image search module 1330 is specifically used for:

[0225] Based on the largest image size among the multiple images to be processed, the multiple images to be processed are sized uniformly to obtain each image to be stitched together.

[0226] The images to be stitched together are stitched together sequentially according to a preset stitching order to obtain the target image;

[0227] Input the target image into the image search model;

[0228] During the size unification process, if the image size of any of the multiple images to be processed is smaller than the largest image size, then based on the largest image size, a region to be filled is determined in at least two surrounding areas of the image to be processed, and a preset pixel value is used to fill the region with pixels.

[0229] In one possible implementation, after inputting the target image into the image search model, the image search module 1330 is further configured to:

[0230] Based on the image search model, the target image is subjected to feature aggregation processing to obtain the comprehensive feature information corresponding to the target image;

[0231] Based on the comprehensive feature information, target feature information with a matching degree greater than the matching threshold is queried from a preset database;

[0232] The object description information corresponding to the target feature information is used as the recognition result of the object to be identified.

[0233] Based on the same inventive concept, see [reference] Figure 14 As shown in the figure, this application provides an inspection device, including:

[0234] Memory 141 is used to store computer programs or instructions;

[0235] Processor 142 is configured to execute a computer program or instructions in memory 141 such that any of the methods described in the above embodiments is performed.

[0236] Image acquisition device 143 is used to acquire images.

[0237] The processor 142 may include one or more central processing units (CPUs) or digital processing units, etc.

[0238] It should be noted that the specific connection medium between the memory 141, processor 142, and image acquisition device 143 is not limited in the embodiments of this application. The embodiments of this application... Figure 14In this diagram, the memory 141, processor 142, and image acquisition device 143 are connected via a bus 144. The connections between other components are merely illustrative and not intended to be limiting. The bus 144 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0239] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when executed by a processor, enables the processor to perform any of the methods described in the above embodiments. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the object recognition method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0240] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0242] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.

[0243] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0244] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An object recognition method, characterized in that, The method, applied to inspection equipment with an image acquisition device, includes: The image acquisition device captures images of the object to be identified from multiple angles within the field of view, thereby obtaining images of the object from multiple angles. The object to be identified is the item carried by the person who needs to be checked by security. For each image angle, the following operations are performed: A rotating target detection algorithm is invoked to detect the target in the image at any of the multiple angles, obtaining the position and angle information of the rotating rectangle that selects the object to be identified in the image at any given angle, wherein the angle information represents the tilt angle of the rotating rectangle; based on the position information, the image of the object to be identified is segmented from the image at any given angle; based on the angle information, the image of the object to be identified is corrected from the tilt angle to a normal position, obtaining the image to be processed, wherein the image to be processed is the normal position image of the object to be identified at any given angle; The obtained multiple images to be processed are made up to the same size, and the images with the same size are stitched together to obtain the target image. The target image is then input into a preset image search model to obtain the recognition result of the object to be identified.

2. The method as described in claim 1, characterized in that, The location information includes multiple coordinate information; the step of segmenting the image of the object to be identified from the image at any angle based on the location information includes: Iterate through each first pixel within the image at any given angle; For each first pixel traversed, the angle values ​​of each angle with the first pixel as the vertex are determined based on the position information and the coordinate information of the currently traversed first pixel. The coordinate information of other points associated with any of the angles is selected from the plurality of coordinate information according to a preset selection rule. If the sum of the determined angle values ​​is less than a preset value, then the first pixel is marked. After the traversal is completed, the image of the object to be identified is segmented from the image at any angle based on each of the marked first pixels.

3. The method as described in claim 1, characterized in that, The step of correcting the image of the object to be identified from the tilt angle to an upright position based on the angle information to obtain the image to be processed includes: Based on the angle information, the image of the object to be identified is rotated according to a preset rotation rule to obtain an intermediate image, wherein the rotation rule includes a rotation center and a rotation direction; Based on the angle information and the coordinate information corresponding to the rotation center, the rotated coordinate information corresponding to each first pixel in the intermediate image is determined; Based on the rotated coordinate information corresponding to each first pixel, a new image coordinate system is constructed, and based on the pixel value corresponding to each first pixel, the pixel value corresponding to each second pixel in the intermediate image in the image coordinate system is determined. The image to be processed is determined based on the pixel value corresponding to each second pixel.

4. The method as described in claim 3, characterized in that, The step of determining the pixel value corresponding to each second pixel in the intermediate image in the image coordinate system based on the pixel values ​​corresponding to each first pixel includes: Traverse each second pixel in the intermediate image within the image coordinate system. For each second pixel traversed, query the pixel value corresponding to the coordinate information of the currently traversed second pixel from the rotated coordinate information corresponding to each first pixel. If the query finds a match, the pixel value of the first pixel will be used as the pixel value of the second pixel. If no match is found, multiple target pixels are selected from the first pixels based on the coordinate information of the second pixel and the rotated coordinate information of each first pixel. The pixel value of the second pixel is determined based on the pixel values ​​of the multiple target pixels, wherein the distance between the target pixel and the second pixel is less than a distance threshold.

5. The method as described in claim 4, characterized in that, Determining the pixel value of the second pixel based on the pixel values ​​of the plurality of target pixels includes: The average pixel value of the plurality of target pixels is used as the pixel value of the second pixel; or The weighted sum of the pixel values ​​of the plurality of target pixels is used as the pixel value of the second pixel, wherein the smaller the distance between the target pixel and the second pixel, the larger the weight.

6. The method according to any one of claims 1-5, characterized in that, The process of unifying the size of the multiple images to be processed includes: Based on the largest image size among the multiple images to be processed, the multiple images to be processed are sized uniformly to obtain each image to be stitched together. The step of stitching together images of uniform size to obtain the target image includes: The images to be stitched together are stitched together sequentially according to a preset stitching order to obtain the target image; During the size unification process, if the image size of any of the multiple images to be processed is smaller than the largest image size, then based on the largest image size, a region to be filled is determined in at least two surrounding areas of the image to be processed, and a preset pixel value is used to fill the region with pixels.

7. The method as described in claim 6, characterized in that, After inputting the target image into the preset image search model, the method further includes: Based on the image search model, the target image is subjected to feature aggregation processing to obtain the comprehensive feature information corresponding to the target image; Based on the comprehensive feature information, target feature information with a matching degree greater than the matching threshold is queried from a preset database; The object description information corresponding to the target feature information is used as the recognition result of the object to be identified.

8. An object recognition device, characterized in that, Applied to inspection equipment with an image acquisition device, the device includes: The acquisition module is used to capture images of the object to be identified from multiple angles within the field of view using the image acquisition device, thereby obtaining images of the object to be identified from multiple angles, wherein the object to be identified is an item carried by a person requiring security inspection; The segmentation module performs the following operations for each angle of the image: It invokes a rotating target detection algorithm to detect the target in the image at any of the multiple angles, obtaining the position and angle information of the rotating rectangle that selects the object to be identified in the image at any angle, wherein the angle information represents the tilt angle of the rotating rectangle; based on the position information, it segments the image of the object to be identified from the image at any angle; based on the angle information, it corrects the image of the object to be identified from the tilt angle to a normal position, obtaining the image to be processed, wherein the image to be processed is the normal position image of the object to be identified at any angle; The image search module is used to unify the size of multiple images to be processed, and then stitch the images to obtain a target image. The target image is then input into a preset image search model to obtain the recognition result of the object to be identified.

9. An inspection device, characterized in that, include: Image acquisition device, used to acquire images; Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-7 is performed.

Images