Commodity Settlement Processing Method, Device, Terminal Device and Storage Medium

Abstract

This application is applicable to the field of computer technology, and provides a commodity settlement processing method, device, terminal device and storage medium. The method includes: obtaining N first images captured from N shooting angles of a settlement area, where N is a positive integer greater than 1; respectively identifying commodities in each first image to obtain preliminary identification results corresponding to the N first images; determining a final identification result according to the obtained N groups of preliminary identification results, where the final identification result includes the categories and quantities of commodities in the settlement area; and settling the commodities in the settlement area according to the final identification result. The above method can improve the accuracy of commodity identification, and further improve the accuracy of commodity settlement.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and particularly relates to a commodity settlement processing method, device, terminal device, and storage medium. Background Art

[0002] With the continuous development of the field of computer vision, image recognition technology has been widely used in the retail industry, and the self-service commodity settlement method has also been gradually optimized. The self-service commodity settlement method based on image recognition technology has emerged, that is, a method in which a computer applies deep learning algorithms to process, analyze, and understand commodity images to identify the types and quantities of commodities.

[0003] Currently, the self-service commodity settlement method is implemented through self-service settlement instruments. When the existing self-service settlement instruments identify multiple commodities placed in the settlement area, commodity missing detections often occur due to partial or complete occlusion of the commodities, reducing the commodity recognition accuracy, and thus resulting in inaccurate commodity settlement. Summary of the Invention

[0004] The embodiments of this application provide a commodity settlement processing method, device, terminal device, and storage medium, which can improve the commodity recognition accuracy, and thus improve the accuracy of commodity settlement.

[0005] In a first aspect, the embodiments of this application provide a commodity settlement processing method, including:

[0006] Obtaining N first images taken of the settlement area from N shooting angles, where N is a positive integer greater than 1;

[0007] Respectively identifying the commodities in each first image to obtain the preliminary recognition results corresponding to each of the N first images;

[0008] Determining a final recognition result according to the obtained N groups of preliminary recognition results, where the final recognition result includes the commodity types and quantities of the commodities in the settlement area;

[0009] Settling the commodities in the settlement area according to the final recognition result.

[0010] In a second aspect, the embodiments of this application provide a commodity settlement processing device, including:

[0011] A first obtaining module, configured to obtain N first images taken of the settlement area from N shooting angles, where N is a positive integer greater than 1;

[0012] A first processing module, configured to respectively identify the commodities in each first image to obtain the preliminary recognition results corresponding to each of the N first images;

[0013] A second processing module, configured to determine a final recognition result according to the obtained N sets of preliminary recognition results, where the final recognition result includes the product categories and quantities of the products in the settlement area;

[0014] A product settlement module, configured to settle the products in the settlement area according to the final recognition result.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, it implements the product settlement processing method according to any one of the above first aspects.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, it implements the product settlement processing method according to any one of the above first aspects.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which when running on a terminal device, causes the terminal device to execute the product settlement processing method according to any one of the above first aspects.

[0018] The beneficial effects of the embodiment of the first aspect of the present application compared with the prior art are as follows:

[0019] By obtaining N first images captured from N shooting angles of the settlement area, where N is a positive integer greater than 1; respectively identifying the products in each first image to obtain the preliminary recognition results corresponding to the N first images; determining the final recognition result according to the obtained N sets of preliminary recognition results, where the final recognition result includes the categories and quantities of the products in the settlement area; and settling the products in the settlement area according to the final recognition result, in this way, by performing product recognition on the images of the settlement area captured from multiple shooting angles, the occurrence of product omission detection can be avoided, thereby improving the accuracy of product recognition and further improving the accuracy of product settlement.

[0020] It can be understood that the beneficial effects of the above second aspect to fifth aspect can refer to the relevant descriptions in the above first aspect, and will not be elaborated here. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is a schematic flowchart of a commodity settlement processing method provided by an embodiment of the present application;

[0023] Figure 2 is Figure 1 a schematic diagram of the specific implementation steps of step 103 in

[0024] Figure 3 It is a schematic structural diagram of a commodity settlement processing device provided by an embodiment of the present application;

[0025] Figure 4 It is a schematic structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0026] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented in order to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0027] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0028] As used in the specification of the present application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0029] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0030] Reference to "one embodiment" or "some embodiments" etc. described in the specification of this application means that in one or more embodiments of this application, specific features, structures or characteristics described in connection with that embodiment are included. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] Figure 1 FIG. shows a schematic flowchart of a commodity settlement processing method provided by this application. Referring to Figure 1 , the detailed description of this commodity calculation processing method is as follows:

[0032] S101, Obtain N first images captured from N shooting angles of a settlement area, where N is a positive integer greater than 1.

[0033] Specifically, the shooting of the settlement area from N shooting angles can be achieved through a camera. Here, the camera can be one, and the shooting position of the camera is moved to achieve image acquisition at different shooting angles; the camera can also be multiple, and multiple cameras are set at different shooting positions, where the shooting angles corresponding to different shooting positions are different, so as to achieve image acquisition at different shooting angles.

[0034] S102, Identify the commodities in each of the first images respectively, and obtain the preliminary recognition results corresponding to the N first images respectively.

[0035] Optionally, the preliminary recognition result of the first image includes the commodity category and similarity of each commodity in the first image.

[0036] Here, the similarity of the commodity refers to the similarity of the corresponding commodity category.

[0037] It should be noted that the identification of the commodities in the first image can be achieved through commodity detection and feature vector extraction. For details, see Embodiment 1.

[0038] S103, Determine the final recognition result according to the obtained N groups of preliminary recognition results, where the final recognition result includes the category and quantity of the commodities in the settlement area.

[0039] It should be noted that since the N first images correspond to images taken from N shooting angles, if the first commodity is partially or completely blocked within the field of view corresponding to one shooting angle, resulting in the failure to recognize the first commodity, while the first commodity is unblocked within the field of view corresponding to another shooting angle, then the first commodity can be recognized. Therefore, the commodities in the N groups of preliminary recognition results recognized after commodity recognition may be the same or different, but they are all commodities within the settlement area. Therefore, it is necessary to comprehensively consider the N groups of preliminary recognition results to determine the final recognition result, which can improve the accuracy of commodity recognition.

[0040] Here, the final recognition result can be obtained by performing matching calculations on the N groups of preliminary recognition results. For the detailed description, see Embodiment 2.

[0041] S104. Settle the commodities within the settlement area according to the final recognition result.

[0042] Specifically, according to the category of the commodities in the final recognition result, obtain the corresponding commodity unit price from the first preset commodity database; calculate the amount to be paid based on the category, unit price, and quantity of the commodities; and complete the commodity calculation according to the amount to be paid.

[0043] It should be noted that the first preset commodity database is established in advance, and this first preset commodity database includes the commodity information of all sold commodities. The commodity information includes contents such as commodity name, commodity barcode, commodity unit price, and commodity weight.

[0044] Embodiment 1

[0045] In a possible implementation manner, the implementation process of step S102 may include:

[0046] S1021. Detect the local images of each commodity in the first image.

[0047] Optionally, step S1021 may specifically include:

[0048] a1. Input the first image into the first detection model to obtain the position information of each commodity in the first image.

[0049] Specifically, the position information of the commodity is the position coordinates of the commodity. It should be noted that the centroid coordinates of the commodity can be calculated according to the position coordinates of the commodity.

[0050] It should be noted that the first detection model is based on a detection model that has been pre-trained. The specific model training steps may include:

[0051] 1) Collect the data set.

[0052] Specifically, a variety of commodities are placed in the settlement area without overlap at multiple angles, and cameras with different shooting angles are used to shoot the settlement area. All commodity categories and position coordinates in the settlement area are marked, and all commodity categories are modified to main commodities.

[0053] 2) Enhance and preprocess the dataset.

[0054] Specifically, perform random scaling, random color jittering, and normalization processing on the dataset.

[0055] 3) Model training.

[0056] Specifically, construct a potential commodity detection model based on the main body detection technology, and use the processed dataset in step 2) to train the constructed potential commodity detection model to obtain the first detection model.

[0057] a2: Detect the local image of each commodity from the first image according to the position information of each commodity.

[0058] Specifically, intercept the commodities in the first image according to the position coordinates of each commodity to obtain the local image of each commodity.

[0059] S1022, obtain the first feature vector of each local image in the first image.

[0060] Here, each local image in the first image can be respectively input into the feature extraction model to obtain the first feature vector of each local image in the first image.

[0061] It should be noted that the feature extraction model is based on a pre-trained model. The specific model training steps can include:

[0062] i) Collect the dataset.

[0063] Specifically, a variety of commodities are placed in the settlement area without overlap at multiple angles, and cameras with different shooting angles are used to shoot the settlement area. All commodity categories and position coordinates in the settlement area are marked; the commodities in the captured image are intercepted according to the position coordinates of the commodities, and the intercepted image is preprocessed.

[0064] ii) Enhance and preprocess the dataset.

[0065] Specifically, perform gray bar addition, random color jittering, random scaling, random erasing, and image mixing processing on the dataset.

[0066] iii) Model training.

[0067] Specifically, a product feature extraction model is constructed, and the constructed product feature extraction model is trained using the processed dataset in step (ii) to obtain a feature extraction model.

[0068] S1023. For each first feature vector, calculate the similarity between the first feature vector and each second feature vector in the first preset product feature library to obtain the target feature vector with the highest similarity to the first feature vector among the second feature vectors.

[0069] It should be noted that the first preset product feature library is established in advance and includes the feature vectors and product categories of all sold products.

[0070] In an optional implementation manner, an index table is established based on all the feature vectors in the first preset product feature library. Each index in the index table corresponds to a second feature vector, and the second feature vector is a feature vector representing a set. The set includes feature vectors of multiple product categories, where the similarities of the feature vectors of the multiple product categories are within a preset range. That is to say, the products corresponding to the multiple product categories are similar products. Then, calculate the similarity between the first feature vector and each second feature vector of the index to obtain the second feature vector with the highest similarity to the first feature vector among the second feature vectors, determine the target index corresponding to the second feature vector with the highest similarity to the first feature vector, and calculate the similarity between the first feature vector and each feature vector in the target index to obtain the target feature vector with the highest similarity to the first feature vector among the feature vectors under the target index. In this way, the first feature vector does not need to be compared one by one with all the feature vectors in the first preset product feature library, which can improve the calculation efficiency and save computing power.

[0071] The second feature vector is a feature vector representing a set. Specifically, the second feature vector can be a feature vector among all the feature vectors in the set, or a central feature vector calculated based on all the feature vectors in the set.

[0072] S1024. Determine the product category of the product corresponding to the first feature vector as the target product category corresponding to the target feature vector, and determine the similarity of the product as the similarity between the target feature vector and the first feature vector.

[0073] Embodiment 2

[0074] See Figure 2 In a possible implementation manner, the implementation process of step S103 may include:

[0075] S1031. Match the reference group with a non-reference group to obtain a first intermediate recognition result. Here, the reference group is any one of the N preliminary recognition results, and the non-reference group is the preliminary recognition results other than the reference group among the N preliminary recognition results.

[0076] Optionally, the preliminary recognition result further includes the centroid coordinates of each commodity in the first image. That is to say, each group of preliminary recognition results includes the commodity category, similarity, and centroid coordinates of each commodity in the first image.

[0077] Here, the reference group and the non-reference group can be matched based on the centroid coordinates of the commodities to obtain a first intermediate result. The first intermediate result includes the recognition results determined by successful matching of the reference group and the non-reference group and the recognition results determined by unsuccessful matching of the reference group and the non-reference group. For the specific implementation process, please refer to the description in Part Three of the embodiment.

[0078] S1032. Update the reference group to the first intermediate recognition result, and match the updated reference group with the next non-reference group until the second intermediate recognition result after matching the last non-reference group with the reference group is obtained.

[0079] It should be noted that this step is a loop execution process. The matching process between the updated reference group and the next non-reference group is similar to the matching process in step S1031. For the specific implementation process, please refer to the description in Part Three of the embodiment.

[0080] S1033. Determine the final recognition result as the second intermediate recognition result.

[0081] Here, the final recognition result includes the quantity of commodities in the settlement area, the centroid coordinates of each commodity, the commodity category, and the similarity.

[0082] Embodiment Three

[0083] In a possible implementation manner, the implementation process of step S1031 may include:

[0084] b1. Perform coordinate transformation on the centroid coordinates of each commodity in the non-reference group to obtain the transformed coordinates of each commodity. Here, the transformed coordinates of each commodity in the non-reference group and the centroid coordinates of each commodity in the reference group belong to the same coordinate system.

[0085] Here, since the N sets of preliminary recognition results correspond to N first images, and the N first images are images taken of the settlement area from N shooting angles, the centroid coordinates of the commodities in each set of preliminary recognition results are based on the coordinate system corresponding to their respective shooting angles. To calculate the distances between the commodities in different sets of preliminary recognition results subsequently, it is necessary to use the coordinate system corresponding to one shooting angle (the set of preliminary recognition results corresponding to this shooting angle is the preliminary recognition results of the reference group), and the centroid coordinates of the commodities in the preliminary recognition results corresponding to other shooting angles are all converted to this coordinate system, so as to achieve the calculation in the same coordinate system.

[0086] Specifically, according to the shooting positions and shooting angles of the cameras corresponding to the shooting of the N first images, a coordinate mapping relationship matrix is determined, and coordinate conversion is realized based on this mapping relationship matrix, where the coordinate mapping relationship matrix is used to represent the coordinate mapping relationships from the first images corresponding to N - 1 non-reference groups to the first image corresponding to the reference group.

[0087] For example, N cameras can be set at different positions above the settlement area. Specifically, one camera is set directly above the settlement area, and the first image taken by this camera is the front view; the other N - 1 cameras are arranged around the camera located directly above the settlement area and are set at different positions, and the first images taken by these N - 1 cameras are the side views. Then, according to the positions and shooting angles of each camera, the mapping relationship matrix from the side view to the front view can be determined.

[0088] b2: Calculate the distances from the converted coordinates of the j-th commodity in the non-reference group to the centroid coordinates of each commodity in the reference group, obtaining multiple distance values, where j is a positive integer less than or equal to M, and M is the number of commodities in the non-reference group.

[0089] It should be noted that j in the j-th commodity starts from 1 and goes up to M. That is to say, each commodity in the non-reference group has to execute steps b2 - b3, or execute steps b2 - b4.

[0090] b3: If there is a target distance among the multiple distance values, and there is exactly one target distance, then generate a commodity matching result with the first target value, the second target value, and the centroid coordinates of the commodity corresponding to the target distance in the reference group, add the commodity matching result to the first intermediate recognition result, and delete the preliminary recognition result of the commodity corresponding to the target distance from the reference group, where the target distance is the distance value that satisfies the preset threshold range and has the smallest value among the multiple distance values; the first target value is the maximum similarity between the first result and the second result; the second target value is the commodity category corresponding to the maximum similarity between the first result and the second result; the first result is the initial recognition result of the j-th commodity in the non-reference group; the second result is the initial recognition result of the commodity corresponding to the target distance in the reference group.

[0091] Here, the target distance is the distance value that satisfies the preset threshold range and has the smallest value among multiple distance values, and there is one and only one such target distance, indicating that the j-th commodity in the non-reference group coincides with a commodity in the reference group within the preset threshold range, that is, within the preset error range, indicating that the commodity in the reference group that coincides with the j-th commodity in the non-reference group within the preset threshold range corresponds to the same commodity as the j-th commodity, that is, the matching is successful.

[0092] It should be noted that deleting the preliminary recognition result of the commodity corresponding to the target distance in the successfully matched reference group from the reference group, one of the purposes is that since the determined commodity matching result has been added to the first intermediate result, deletion can save storage space; the second purpose is to improve the matching calculation efficiency of the (j + 1)-th commodity in the subsequent non-reference group and the commodities in the reference group, and save computing power.

[0093] It should be noted that when there is the above-mentioned target distance among multiple distance values, but the number of target distances is more than one, it means that the j-th commodity in the non-reference group coincides with more than one candidate commodity in the reference group within the preset threshold range, that is, within the preset error range, that is, it is impossible to determine which one of the candidate commodities the j-th commodity corresponds to, and no processing is done at this time.

[0094] b4: If all multiple distance values are distance values outside the preset threshold range, then add the preliminary recognition result of the j-th commodity in the non-reference group to the first intermediate recognition result;

[0095] Here, all multiple distance values are distance values outside the preset threshold range, indicating that the distance between the j-th commodity in the non-reference group and any commodity in the reference group is not within the preset threshold range, that is, there is no position coincidence within the preset error range, indicating that the j-th commodity in the non-reference group is a commodity in the settlement area that has not been recognized in the first image corresponding to the reference group, so the preliminary recognition result of this j-th commodity also needs to be added to the first intermediate recognition result to ensure the accuracy of commodity recognition.

[0096] b5: After each commodity in the non-reference group participates in the result matching calculation, if there are still preliminary recognition results of the remaining commodities in the reference group, then add the preliminary recognition results of the remaining commodities to the first intermediate recognition result.

[0097] Here, after each commodity in the non-reference group participates in the result matching calculation, if there are still preliminary recognition results of the remaining commodities in the reference group, it means that there are still commodities in the reference group that have not been successfully matched, but they are also commodities in the settlement area. At this time, the preliminary recognition results of the remaining commodities also need to be added to the first intermediate recognition result to ensure the accuracy of commodity recognition.

[0098] The following uses an example to illustrate the implementation process of step S103.

[0099] It should be noted that in this example, N cameras are set at different positions above the settlement area. Specifically, one camera is set directly above the settlement area, and the first image captured by this camera is the front view; the other N - 1 cameras are arranged around the camera directly above the settlement area and are located at different positions. The first images captured by these N - 1 cameras are side views. The specific commodity matching process is as follows:

[0100] 1) Determine the mapping relationship matrix from the side view to the front view according to the positions and shooting angles of each camera, and convert the centroid coordinates of each commodity in the commodity recognition result corresponding to the side view according to this mapping relationship matrix to obtain the centroid conversion coordinates.

[0101] 2) After obtaining the conversion coordinates of the commodities in the recognition result corresponding to the side view, the recognition result corresponding to the front view can be named list0, and the recognition result of one of the side views can be named list1.

[0102] 3) Initialize an empty commodity matching list result_list to save the results of this commodity matching.

[0103] 4) Take out the recognition result of the first commodity from list1, and calculate the distance value from the centroid conversion coordinates of the commodity in this recognition result to the centroid coordinates of all commodities in list0.

[0104] Specifically, if the distance from the centroid coordinates of a commodity in list0 to the centroid conversion coordinates of the first commodity in list1 is within the preset error range and the minimum distance value is unique, then take the centroid coordinates of the recognition result of the target commodity in list0, the commodity category and similarity in the recognition result with a larger similarity between the recognition result of the first commodity in list1 and the recognition result of the target commodity, and form a commodity matching result with the above three values, add it to result_list, and delete the recognition result of the target commodity from list0, where the target commodity is the commodity in list0 related to the minimum distance value.

[0105] If the distance from the centroid coordinates of a commodity in list0 to the centroid conversion coordinates of the first commodity in list1 is within the preset error range and the minimum distance value is not unique, no processing is performed.

[0106] If the distances between the centroid coordinates of all commodities in list0 and the centroid coordinates of the first commodity in list1 all exceed the preset error range, then add the recognition result of the first commodity in list1 to result_list.

[0107] 5) Then continue to take out the recognition result of the second product from list1, and repeat the above step 4) until the recognition result of each product in list1 has completed a matching calculation.

[0108] 6) If list0 is not empty after the recognition result of each product in list1 has completed a matching calculation, then add the recognition results of all the remaining products in list0 to result_list.

[0109] 7) Replace list0 with result_list in 6), continue to take the recognition result of the next side view, name it list1, and then repeat the above steps 4) to 6) until the matching of the recognition results of the products in each side view is completed, and the final product matching result result_list is obtained.

[0110] In order to further improve the accuracy of product settlement, in a possible implementation, the implementation process of step S104 may include:

[0111] Step S1041, obtain the weight of each product in the final recognition result from the first preset product database, and calculate the total weight of the first products.

[0112] As can be seen from the above, the first preset product database includes the product information of each sold product. According to the product information of each product in the final recognition result in the first preset product database, the weight corresponding to each product is obtained, and the weights of all the products in the final recognition result are summed to calculate the total weight of the first products.

[0113] Step S1042, obtain the total weight of the second products in the settlement area measured by the weight sensing device.

[0114] It should be noted that the weight sensing device may include a weight sensor and a weight transmission and calculation program. Among them, the weight sensor is used to measure the total weight of the products in the settlement area, which is recorded as the total weight of the second products.

[0115] Step S1043, determine whether the difference between the total weight of the first products and the total weight of the second products is within the preset error range.

[0116] Step S1044, if the difference between the total weight of the first products and the total weight of the second products is within the preset error range, then settle the products in the settlement area according to the final recognition result.

[0117] Here, the difference between the total weight of the first products and the total weight of the second products being within the preset error range indicates that the total weight of the products calculated by image recognition is approximately equal to the total weight of the products measured by the weight sensing device, which indirectly proves that the image recognition this time is accurate.

[0118] After step S1043, the above method may further include:

[0119] If the difference between the total weight of the first commodity and the total weight of the second commodity is outside the preset error range, a first prompt message is generated, where the first prompt message is used to prompt the customer to re-place the commodity to be settled in the settlement area.

[0120] After that, if the customer re-places the commodity to be settled in the settlement area according to the first prompt message, and after the placement duration exceeds the preset duration, steps S101 to S104 are repeatedly executed.

[0121] As can be seen from the above optional embodiments, in step S1021, detecting the local image of each commodity in the first image is implemented by the first detection model, and in the first detection model, the commodity category and the position coordinates of the commodity can be detected. The commodity categories that the first detection model can detect are related to the commodity categories in the dataset corresponding to the pre-trained model. If the commodity category is not the commodity category corresponding to the first detection model, that is, a new commodity category, the first detection model cannot detect the commodity category of the commodity, but can detect the position coordinates of the commodity. If it is necessary for the first detection model to detect the new commodity category, the first detection model needs to be re-trained, which not only takes a long time but also has poor practicability. To solve the above problems, in a possible implementation manner, the above method of the embodiment of the present application may further include:

[0122] S201: Obtain the basic information of the first commodity of the new commodity category, and add the basic information to the second preset commodity database.

[0123] It should be noted that the basic information of the first commodity includes the commodity name, commodity barcode, commodity unit price, commodity specification, commodity inventory, commodity weight, etc. of the first commodity.

[0124] S202: Obtain the second images of the first commodity taken from N shooting angles at different positions of the first commodity in the settlement area multiple times.

[0125] Here, each time the first commodity is at a position in the settlement area, N second images of the settlement area taken from N shooting angles need to be obtained.

[0126] S203: When the number of the second images reaches the first preset number, detect the local image of the first commodity in the first preset number of second images.

[0127] Here, the number of second images reaching the first preset number can be achieved by setting the number of times of obtaining the second images. For example, if the number of second images does not reach the first preset number, a second prompt message is generated, and the second prompt message is used to prompt the user to adjust the placement position of the first commodity. After the weight sensing device completes a state change, that is, after the weight sensing state changes from weight stability to weight instability and then to weight recovery stability, the second images obtained by photographing the first commodity at the adjusted position from N shooting angles are acquired.

[0128] When the number of second images processed through step S202 reaches the second preset number but does not reach the first preset number, data augmentation can be used to increase the number of second images to the first preset number.

[0129] It should be noted that the first detection model mentioned above can continue to be used to detect the partial images of the first commodity in the second images of the first preset number, and there is no need to retrain the detection model. Only the position of the commodity needs to be detected by the first detection model.

[0130] S204: Obtain the third feature vectors of the partial images of the first commodity of the first preset number.

[0131] Here, the third feature vectors of the partial images of the first commodity of the first preset number can be obtained through the above-mentioned feature extraction model.

[0132] S205: Add the commodity category and the third feature vectors of the first commodity to the second preset commodity feature library.

[0133] In this way, through the method in the above implementation manner, there is no need to perform deep learning model training again, which can reduce the operation difficulty of commodity entry, greatly reduce the commodity entry time, bring convenience to merchants, and has high practicability.

[0134] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0135] Corresponding to the commodity settlement processing method described in the above embodiments, FIG. 3 shows a structural block diagram of a commodity settlement device provided by an embodiment of the present application. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0136] Refer to Figure 3 and the device 300 may include: a first acquisition module 310, a first processing module 320, a second processing module 330, and a commodity settlement module 340.

[0137] Among them, the first acquisition module 310 is configured to acquire N first images captured from N shooting angles of the settlement area, where N is a positive integer greater than 1.

[0138] The first processing module 320 is configured to respectively identify the commodities in each first image, and obtain preliminary identification results corresponding to each of the N first images.

[0139] The second processing module 330 is configured to determine a final identification result according to the obtained N groups of preliminary identification results, where the final identification result includes the commodity categories and quantities of the commodities in the settlement area.

[0140] The commodity settlement module 340 is configured to settle the commodities in the settlement area according to the final identification result.

[0141] Optionally, the preliminary identification result of the first image includes the commodity category and similarity of each commodity in the first image; correspondingly, the first processing module 320 may include:

[0142] The first detection unit is configured to detect the local images of each commodity in the first image.

[0143] The first acquisition unit is configured to acquire the first feature vectors of each local image in the first image.

[0144] The calculation unit is configured to calculate the similarity between each first feature vector and each second feature vector in the first preset commodity feature library for each first feature vector, and obtain the target feature vector with the highest similarity to the first feature vector among the second feature vectors.

[0145] The first processing unit is configured to determine that the commodity category of the commodity corresponding to the first feature vector is the target commodity category corresponding to the target feature vector, and determine that the similarity of the commodity is the similarity between the target feature vector and the first feature vector.

[0146] In an optional implementation manner, the first detection unit is specifically configured to:

[0147] Input the first image into the first detection model to obtain the position information of each commodity in the first image; detect the local image of each commodity from the first image according to the position information of each commodity.

[0148] In an optional implementation manner, the second processing module 330 may include:

[0149] The first matching unit is configured to match a reference group with a non-reference group to obtain a first intermediate identification result, where the reference group is any one of the N groups of preliminary identification results, and the non-reference group is the preliminary identification results other than the reference group among the N groups of preliminary identification results;

[0150] A second matching unit, configured to update the reference group to a first intermediate recognition result, and match the updated reference group with the next non-reference group until a second intermediate recognition result after matching the last non-reference group with the reference group is obtained.

[0151] A second processing unit, configured to determine that the final recognition result is the second intermediate recognition result.

[0152] Optionally, the preliminary recognition result further includes the centroid coordinates of each commodity in the first image; correspondingly, the first matching unit is specifically configured to:

[0153] Perform coordinate transformation on the centroid coordinates of each commodity in the non-reference group to obtain the transformed coordinates of each commodity, where the transformed coordinates of each commodity in the non-reference group and the centroid coordinates of each commodity in the reference group belong to the same coordinate system;

[0154] Calculate the distances from the transformed coordinates of the j-th commodity in the non-reference group to the centroid coordinates of each commodity in the reference group to obtain a plurality of distance values, where j is a positive integer less than or equal to M, and M is the number of commodities in the non-reference group;

[0155] If there is a target distance among the plurality of distance values, and there is only one target distance that satisfies the first preset condition, generate a commodity matching result with the first target value, the second target value, and the centroid coordinates of the commodity corresponding to the target distance in the reference group, add the commodity matching result to the first intermediate recognition result, and delete the preliminary recognition result of the commodity corresponding to the target distance from the reference group, where the target distance is the distance value that satisfies the preset threshold range and has the smallest value among the plurality of distance values; the first target value is the maximum similarity between the first result and the second result; the second target value is the commodity category corresponding to the maximum similarity between the first result and the second result; the first result is the initial recognition result of the j-th commodity in the non-reference group; the second result is the initial recognition result of the commodity corresponding to the target distance in the reference group.

[0156] If all the plurality of distance values are distance values outside the preset threshold range, add the preliminary recognition result of the j-th commodity in the non-reference group to the first intermediate recognition result.

[0157] After each commodity in the non-reference group participates in the result matching calculation, if there are still preliminary recognition results of the remaining commodities in the reference group, add the preliminary recognition results of the remaining commodities to the first intermediate recognition result.

[0158] In a possible implementation manner, the commodity settlement module 340 may specifically be configured to:

[0159] Obtain the weight of each commodity in the final recognition result from the first preset commodity database, and calculate to obtain the total weight of the first commodity.

[0160] Obtain the total weight of the second commodity in the settlement area measured by the weight sensing device.

[0161] Determine whether the difference between the total weight of the first commodity and the total weight of the second commodity is within a preset error range.

[0162] If the difference between the total weight of the first commodity and the total weight of the second commodity is within the preset error range, then settle the commodities in the settlement area according to the final recognition result.

[0163] In a possible implementation manner, the device according to the embodiment of the present application may further include:

[0164] A second obtaining module, configured to obtain the basic information of the first commodity of the new commodity category and add the basic information to the second preset commodity database.

[0165] A third obtaining module, configured to obtain the second images of the first commodity taken from N shooting angles when the first commodity is at different positions in the settlement area multiple times.

[0166] A detection module, configured to detect the partial images of the first commodity in the second images of the first preset quantity when the quantity of the second images reaches the first preset quantity.

[0167] A fourth obtaining module, configured to obtain the third feature vectors of the partial images of the first commodity of the first preset quantity.

[0168] A third processing module, configured to add the commodity category and the third feature vectors of the first commodity to the second preset commodity feature library.

[0169] It should be noted that the information interaction, execution process, etc. between the above-mentioned device / units, due to being based on the same concept as the method embodiment of the present application, for the specific functions and the technical effects brought, reference may be specifically made to the method embodiment part, and details are not described herein again.

[0170] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here.

[0171] The embodiment of this application also provides a commodity settlement processing system, including a commodity settlement processing device, which is used to implement the steps in any of the foregoing method embodiments.

[0172] Here, the commodity settlement processing system further includes an image acquisition module, a weight sensing module, a human-computer interaction module, a settlement printing module, a voice playback module, and a data storage module.

[0173] Among them, the image acquisition module may include one or more cameras and an image acquisition and preprocessing program, which collects commodity images in the settlement area from multiple shooting angles and transmits them to the commodity settlement processing device to prevent situations such as missed inspection due to commodity occlusion.

[0174] The weight sensing module includes a weight sensor and a weight transmission and calculation program, which transmits the weight of the settlement area to the commodity settlement processing device, so as to judge whether the difference between the actual settlement commodity weight and the commodity weight obtained after detection and matching is within the error range, improving the settlement accuracy.

[0175] The human-computer interaction module includes a touch display device and an automatic settlement application software. Users can view the placement of commodities in the settlement area covered by the camera in the automatic settlement application interface through the touch display screen and make various operation selections.

[0176] The settlement printing module includes a printer / POS machine and a settlement list calculation and printing program. According to the settlement list transmitted by the commodity settlement processing device, it completes the automatic settlement and receipt printing operations.

[0177] The voice playback module includes a voice player and a voice file calling and playback program. It plays various voice prompts according to the instructions of the commodity settlement processing device.

[0178] The data storage module includes a storage device and a read / write operation program for the storage device, which is used to store relevant data such as the first detection model, the feature extraction model, the preset commodity database, and the preset commodity feature library, and cooperate with the commodity settlement processing device to complete data calling and data storage involved in the settlement process.

[0179] An embodiment of the present application also provides a terminal device. Refer to Figure 4 , the terminal device 400 may include: at least one processor 410, a memory 420, and a computer program stored in the memory 420 and executable on the at least one processor 410. When the processor 410 executes the computer program, it implements the steps in any of the above method embodiments, such as Figure 4 the steps S101 to S104 in the illustrated embodiment. Alternatively, when the processor 410 executes the computer program, it implements the functions of each module / unit in the above device embodiments, such as Figure 3 the functions of the illustrated modules 310 to 340.

[0180] Exemplarily, the computer program may be divided into one or more modules / units. One or more modules / units are stored in the memory 420 and executed by the processor 410 to complete the present application. The one or more modules / units may be a series of computer program segments capable of performing specific functions, and this program segment is used to describe the execution process of the computer program in the terminal device 400.

[0181] Those skilled in the art can understand that Figure 4 this is only an example of the terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine some components, or different components, such as input / output devices, network access devices, buses, etc.

[0182] The processor 410 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0183] The memory 420 can be an internal storage unit of the terminal device or an external storage device of the terminal device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. The memory 420 is used to store the computer program and other programs and data required by the terminal device. The memory 420 can also be used to temporarily store the data that has been output or will be output.

[0184] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of easy representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0185] The commodity settlement processing method provided by the embodiments of this application can be applied to terminal devices such as computers, tablet computers, laptop computers, netbooks, personal digital assistants (PDAs), etc. The embodiments of this application do not impose any restrictions on the specific types of terminal devices.

[0186] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0187] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0188] In the embodiments provided in the present application, it should be understood that the disclosed terminal devices, apparatuses, and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0191] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by one or more processors, the steps of the above-mentioned method embodiments can be implemented.

[0192] Similarly, as a computer program product, when the computer program product runs on a terminal device, the terminal device can execute the steps in the above-mentioned method embodiments.

[0193] Among them, the computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0194] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing commodity settlement, characterized in that, it includes: obtaining N first images captured from N shooting angles of a settlement area, where N is a positive integer greater than 1; respectively identifying the commodities in each first image to obtain N groups of preliminary identification results, and the N groups of preliminary identification results include the preliminary identification results corresponding to the N first images respectively; performing coordinate transformation on the centroid coordinates of each commodity in the non-reference group to obtain the transformation coordinates of each commodity, where the reference group is any one of the N groups of preliminary identification results, and the non-reference group is the preliminary identification results other than the reference group in the N groups of preliminary identification results; calculating the distances from the transformation coordinates of the j-th commodity in the non-reference group to the centroid coordinates of each commodity in the reference group to obtain a plurality of distance values, where j is a positive integer less than or equal to M, and M is the number of commodities in the non-reference group; if there is a target distance among the plurality of distance values and there is only one target distance, then generating a commodity matching result with a first target value, a second target value and the centroid coordinates of the commodity corresponding to the target distance in the reference group, and adding the commodity matching result to the first intermediate identification result, where the target distance is the distance value that satisfies a preset threshold range and has the smallest value among the plurality of distance values; the first target value is the maximum similarity between a first result and a second result; the second target value is the commodity category corresponding to the maximum similarity between the first result and the second result; the first result is the initial identification result of the j-th commodity in the non-reference group; the second result is the initial identification result of the commodity corresponding to the target distance in the reference group; obtaining a final identification result according to the first intermediate identification result, and the final identification result includes the commodity category and quantity of the commodities in the settlement area; settling the commodities in the settlement area according to the final identification result.

2. The method for processing commodity settlement according to claim 1, characterized in that, the preliminary identification result of the first image includes the commodity category and similarity of each commodity in the first image; the step of respectively identifying the commodities in each first image to obtain the preliminary identification results corresponding to the N first images respectively includes: detecting the local images of each commodity in the first image; obtaining the first feature vector of each local image in the first image; for each of the first feature vectors, calculating the similarity between the first feature vector and each second feature vector in a first preset commodity feature library to obtain a target feature vector with the highest similarity to the first feature vector among the second feature vectors; determining the commodity category of the commodity corresponding to the first feature vector as the target commodity category corresponding to the target feature vector, and determining the similarity of the commodity as the similarity between the target feature vector and the first feature vector.

3. The method for processing commodity settlement according to claim 2, characterized in that, the step of detecting the local images of each commodity in the first image includes: Input the first image into a first detection model to obtain the position information of each commodity in the first image; Detect the local image of each commodity from the first image according to the position information of each commodity.

4. The commodity settlement processing method according to claim 2, characterized in that, The obtaining the final recognition result according to the first intermediate recognition result includes: Updating the reference group to the first intermediate recognition result, and matching the updated reference group with the next non-reference group until a second intermediate recognition result after matching the last non-reference group with the reference group is obtained; Determining that the final recognition result is the second intermediate recognition result.

5. The commodity settlement processing method according to claim 4, characterized in that, The preliminary recognition result further includes the centroid coordinates of each commodity in the first image; wherein, the conversion coordinates of each commodity in the non-reference group and the centroid coordinates of each commodity in the reference group belong to the same coordinate system; The if there is a target distance among the multiple distance values, and there is only one target distance, then generating a commodity matching result from the first target value, the second target value and the centroid coordinates of the commodity corresponding to the target distance in the reference group, and adding the commodity matching result to the first intermediate recognition result, includes: If there is a target distance among the multiple distance values, and there is only one target distance, then generating a commodity matching result from the first target value, the second target value and the centroid coordinates of the commodity corresponding to the target distance in the reference group, adding the commodity matching result to the first intermediate recognition result, and deleting the preliminary recognition result of the commodity corresponding to the target distance from the reference group; The method further includes: If all the multiple distance values are distance values outside the preset threshold range, then adding the preliminary recognition result of the j-th commodity in the non-reference group to the first intermediate recognition result; After each commodity in the non-reference group participates in the result matching calculation, if there are still preliminary recognition results of the remaining commodities in the reference group, then adding the preliminary recognition results of the remaining commodities to the first intermediate recognition result.

6. The commodity settlement processing method according to claim 1, characterized in that, The settling the commodities in the settlement area according to the final recognition result includes: Obtaining the weight of each commodity in the final recognition result from a first preset commodity database, and calculating to obtain a first total commodity weight; Obtaining a second total commodity weight in the settlement area measured by a weight sensing device; Judging whether the difference between the first total commodity weight and the second total commodity weight is within a preset error range; If the difference between the first total commodity weight and the second total commodity weight is within the preset error range, then settle the commodities in the settlement area according to the final recognition result.

7. The commodity settlement processing method according to claim 1, characterized in that, The method further includes: Obtaining the basic information of the first commodity of the newly added commodity category, and adding the basic information to a second preset commodity database; Obtain second images of the first commodity taken from the N shooting angles when the first commodity is located at different positions in the settlement area multiple times; When the number of the second images reaches a first preset number, detect local images of the first commodity in the second images of the first preset number; Obtain third feature vectors of the local images of the first commodity of the first preset number; Add the commodity category of the first commodity and the third feature vectors to a second preset commodity feature library.

8. A commodity settlement processing device Characterized in that It includes: A first acquisition module, configured to acquire N first images obtained by shooting a settlement area from N shooting angles, where N is a positive integer greater than 1; A first processing module, configured to respectively identify commodities in each of the first images to obtain preliminary identification results corresponding to the N first images; A second processing module, configured to perform coordinate transformation on the centroid coordinates of each commodity in the non-reference group to obtain the transformed coordinates of each commodity, where the reference group is any one of the N groups of preliminary identification results, and the non-reference group is the preliminary identification results in the N groups of preliminary identification results except the reference group; calculate the distances from the transformed coordinates of the j-th commodity in the non-reference group to the centroid coordinates of each commodity in the reference group to obtain a plurality of distance values, where j is a positive integer less than or equal to M, and M is the number of commodities in the non-reference group; if there is a target distance among the plurality of distance values, and there is only one target distance, then generate a commodity matching result with a first target value, a second target value, and the centroid coordinates of the commodity corresponding to the target distance in the reference group, and add the commodity matching result to the first intermediate identification result, where the target distance is the distance value that satisfies a preset threshold range and has the smallest numerical value among the plurality of distance values; the first target value is the maximum similarity between a first result and a second result; the second target value is the commodity category corresponding to the maximum similarity between the first result and the second result; the first result is the initial identification result of the j-th commodity in the non-reference group; the second result is the initial identification result of the commodity corresponding to the target distance in the reference group; obtain a final identification result according to the first intermediate identification result, and the final identification result includes the commodity categories and quantities of the commodities in the settlement area; A commodity settlement module, configured to settle the commodities in the settlement area according to the final identification result.

9. A terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, Characterized in that When the processor executes the computer program, it implements the commodity settlement processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, the computer-readable storage medium stores a computer program, Characterized in that When the computer program is executed by a processor, it implements the commodity settlement processing method according to any one of claims 1 to 7.

Images