Classification support system, classification support method, and program

Abstract

To provide a classification support system, classification support method, and program that can suppress variations in the accuracy of determining the classification number to which a product belongs. [Solution] The classification support system according to this disclosure is a classification support system that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchy in a tariff classification number having a hierarchical structure, and comprises: an acquisition unit that acquires an image of the target product; a first input unit that inputs the image acquired by the acquisition unit to a first trained model which is generated for each of multiple classifications included in a higher hierarchy located above the predetermined hierarchy and outputs classification information to which the product corresponding to the image input at the predetermined hierarchy belongs when an image of the product is input; and a first output unit that outputs classification information at the predetermined hierarchy corresponding to the image input by the first input unit from the first trained model.

Description

[Technical Field]

[0001] The present disclosure relates to a classification support system, a classification support method, and a program. [Background Art]

[0002] Technologies for determining a classification number to which a product belongs have been proposed. For example, Patent Document 1 discloses a technology for determining information related to product classification for a target product. An information processing apparatus according to Patent Document 1 determines a classification number of a product based on first product information about the product. Here, the first product information is, for example, invoice information about the target product. The invoice information includes an invoice number, date, product name, quantity, unit, unit price, amount, supplier information, and the like. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2022-054527 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] The information processing apparatus according to Patent Document 1 determines the classification number to which the target product belongs based on information described in a document such as an invoice. Here, the degree of detail in the description of the target product described in a document such as an invoice may vary depending on the creator of the document. Therefore, when attempting to determine the classification number to which the target product belongs using a document such as an invoice, there is a risk that variation may occur in the accuracy of the determination.

[0005] The present disclosure has been made to solve such problems, and an object of the present disclosure is to provide a classification support system, a classification support method, and a program capable of suppressing variation in the determination accuracy of a classification number to which a product belongs. [Means for Solving the Problem]

[0006] The classification support system relating to this disclosure is a classification support system that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchy in a tariff classification number having a hierarchical structure, and comprises: an acquisition unit that acquires an image of the target product; a first input unit that inputs the image acquired by the acquisition unit to a first trained model that, when an image of the product is input, outputs classification information to which the product corresponding to the input image belongs at the predetermined hierarchy; and a first output unit that outputs classification information at the predetermined hierarchy corresponding to the image input by the first input unit from the first trained model, wherein the first trained model is generated for each of the multiple classifications included in a higher hierarchy located above the predetermined hierarchy, and the first input unit inputs the image to the first trained model generated for the classification in the higher hierarchy to which the target product belongs.

[0007] The classification support method relating to this disclosure is a classification support method in which a computer determines the classification to which a target product belongs from among multiple classifications of a predetermined hierarchy in a tariff classification number having a hierarchical structure, and inputs the acquired image of the target product to a first trained model which is generated for the classification in the higher hierarchy to which the target product belongs from among multiple classifications included in a higher hierarchy located above the predetermined hierarchy, and outputs classification information to which the product corresponding to the input image belongs in the predetermined hierarchy when an image of the product is input, and outputs classification information in the predetermined hierarchy corresponding to the input image of the product from the first trained model.

[0008] The program relating to this disclosure is a program that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchy in a tariff classification number having a hierarchical structure, and causes a computer to perform the following steps: input the acquired image to a first trained model which is generated for the classification in the higher hierarchy to which the target product belongs, among multiple classifications included in a higher hierarchy located above the predetermined hierarchy, and outputs classification information to which the product corresponding to the input image belongs at the predetermined hierarchy when an image of the product is input; and output classification information at the predetermined hierarchy corresponding to the input image from the first trained model. [Effects of the Invention]

[0009] This disclosure provides a classification support system, a classification support method, and a program that can suppress variations in the accuracy of determining the classification number to which a product belongs. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing the configuration of the classification support system related to this disclosure. [Figure 2] Figure 2 is a flowchart showing an example of the flow of a classification support method using a classification support system. [Figure 3] Figure 3 is a block diagram showing the configuration of the classification support system related to this disclosure. [Figure 4] Figure 4 is a block diagram showing the configuration of the higher-level classification decision unit. [Figure 5] Figure 5 is a block diagram showing the configuration of the subclassification determination unit. [Figure 6] Figure 6 is a block diagram showing the configuration of the classification support section. [Figure 7] Figure 7 shows an example of the hardware configuration of a classification support system. [Modes for carrying out the invention]

[0011] (Embodiment 1) Embodiment 1 of the present disclosure will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of the classification support system 1 according to the present disclosure. The classification support system 1 is typically one or more computer devices that operate by a processor executing a program stored in memory. In other words, the classification support system 1 may consist of one computer device or two or more computer devices. For example, the classification support system 1 is one or more server devices.

[0012] Classification support system 1 is a system that determines the classification to which a target product belongs from among multiple classifications at a predetermined level in the hierarchical structure of the tariff classification number. The tariff classification number is also called the HS (Harmonized Commodity Description and Coding System) code. The tariff classification number is a code determined for each attribute of a product based on the "International Convention on a Harmonized System for the Name and Classification of Commodities". The tariff classification number consists of a 6-digit number common to all countries in the world, with the first two digits called the "Chapter", the first four digits called the "Heading", and the first six digits called the "Subheading". In addition, the 6th digit and below of the tariff classification number are determined by a different domestic sub-number in each country, and in Japan this is a 3-digit number. The tariff classification number may refer to the 6-digit number, or it may refer to the number including the domestic sub-number (9 digits in Japan).

[0013] Tariff classification numbers have a hierarchical structure consisting of "Chapter," "Heading," and "Subheading." "Chapter," "Heading," and "Subheading" are referred to as "hierarchies." A "Chapter" is a higher hierarchy than "Heading" and "Subheading," and a "Heading" is a higher hierarchy than "Subheading." Similarly, a "Subheading" is a lower hierarchy than "Chapter" and "Heading," and a "Heading" is a lower hierarchy than "Chapter." Furthermore, "Chapter" and "Heading" can be referred to as the "parent classifications" of "Heading" and "Subheading," respectively. Similarly, "Heading" and "Subheading" can be referred to as the "child classifications" of "Chapter" and "Heading," respectively. Lower-level classifications are subdivisions of each higher-level classification. That is, for example, a "Heading" is one of several classifications that subdivide the classifications of each "Chapter." Here, a given "Chapter" may have only one "Heading" below it, and a given "Heading" may have only one "Subheading" below it.

[0014] Classification support system 1 determines the classification to which the subject product belongs from among multiple classifications at a predetermined hierarchical level in the tariff classification number. Here, the "predetermined hierarchical level" to which classification support system 1 determines the classification is a hierarchical level that has another hierarchical level above it. In other words, classification support system 1 is a system that determines the classification at any lower hierarchical level in the tariff classification number. Specifically, classification support system 1 determines the "heading" or "subheading" to which the subject product belongs. Classification support system 1 may determine both the "heading" and "subheading" classifications.

[0015] The classification support system 1 comprises an acquisition unit 11, an input unit 12, and an output unit 13. The acquisition unit 11, input unit 12, and output unit 13 are typically software or modules whose processing is performed by a processor executing a program stored in memory. The acquisition unit 11, input unit 12, and output unit 13 may also be hardware such as circuits or chips. That is, the acquisition unit 11, input unit 12, and output unit 13 may each be composed of different computer devices.

[0016] At least the acquisition unit 11 and the input unit 12, the input unit 12 and a trained model described later, and the trained model and the output unit 13 are connected to enable data communication. These data communications may be performed via a wired connection or a wireless connection. Data communication may be performed within the same computer device, may be performed via an Internet line, or may be performed using short-range wireless communication technology. Any type of communication protocol can be used for data communication.

[0017] The acquisition unit 11 acquires a captured image of a target product for which classification in a predetermined hierarchy of tariff classification numbers is to be determined. Here, the product refers to a product for which a tariff classification number can be determined by any method. Further, the captured image of the product is typically an image obtained by directly capturing the product for which the tariff classification number is to be determined, but the captured image is not limited thereto. For example, when the target product can be mass-produced, the captured image may be an image obtained by capturing another mass-produced product.

[0018] The input unit 12 inputs the captured image acquired by the acquisition unit 11 to a trained model that outputs classification information indicating which classification in a predetermined hierarchy of tariff classification numbers the product corresponding to the input captured image belongs to when a captured image of the product is input. Here, the trained model may also be referred to as a "first trained model". Further, the input unit 12 may also be referred to as a "first input unit". The classification information output by the trained model includes at least the classification number of the predetermined hierarchy that the trained model determines the target product belongs to.

[0019] The trained model is a model generated for each of a plurality of classifications included in an upper layer positioned above a predetermined layer. For example, in a case where it is intended to determine the "item" of a target product, it is assumed that the "class", which is the upper layer, is classified into "Class A", "Class B", and "Class C". In this case, the trained model is generated for each of "Class A", "Class B", and "Class C". Then, the input unit 12 inputs the captured image acquired by the acquisition unit 11 into the trained model generated for the classification in the upper layer to which the target product belongs. For example, when it is intended to determine the "item" of a target product belonging to "Class A", the input unit 12 inputs the captured image into the trained model generated for "Class A".

[0020] The trained model is a model generated by a machine learning algorithm based on a neural network (NN) such as a CNN (Convolutional Neural Network), for example. Alternatively, the trained model may be generated using a decision tree machine learning algorithm.

[0021] The output unit 13 outputs, from the trained model, classification information in the predetermined layer corresponding to the captured image input by the input unit 12. The output unit 13 may also be referred to as a "first output unit". The output unit 13 outputs the classification information output by the trained model. The output unit 13 may output the classification information to a predetermined user, or may output the classification information to another computer device. Here, the user to whom the output unit 13 outputs the classification information is, for example, a customs broker.

[0022] Next, the flow of the classification support method by the classification support system 1 will be explained. Figure 2 is a flowchart showing an example of the flow of the classification support method by the classification support system 1. First, the acquisition unit 11 acquires an image of the target product to be classified at a predetermined level (S101). Next, the input unit 12 inputs the image acquired by the acquisition unit 11 to a trained model that outputs classification information to which the product corresponding to the input image belongs at a predetermined level when an image of the product is input (S102). After that, the output unit 13 outputs the classification information at a predetermined level corresponding to the image input by the acquisition unit 11 from the trained model (S103).

[0023] Thus, in this embodiment, the classification support system 1 can output classification information to which the target product belongs at a predetermined hierarchy. Specifically, the classification support system 1 outputs classification information using a trained model generated using a NN-based or decision tree-based machine learning algorithm. The trained model used by the classification support system 1 outputs classification information by inputting an image of the target product. As a result, the classification support system 1 can suppress variations in the accuracy of determining the classification number to which the product belongs.

[0024] (Embodiment 2) This second embodiment is a specific example of the first embodiment described above. Figure 3 is a block diagram showing the configuration of the classification support system 2 according to this disclosure. The classification support system 2 comprises an acquisition unit 21, a storage unit 22, a higher-level classification determination unit 23, a lower-level classification determination unit 24, a classification assistance unit 25, and a display unit 26. The components of the classification support system 2 are typically software or modules whose processing is performed by a processor executing a program stored in memory. The classification support system 2 may also be hardware such as a circuit or a chip. That is, the components of the classification support system 2 may each be different computer devices. For example, the classification support system 2 is one or more server devices. Hereafter, for the sake of clarity, explanations that overlap with the first embodiment will be omitted as appropriate.

[0025] In the classification support system 2, predetermined components are connected to each other in a manner that allows for data communication as needed. Data communication between components may be performed via wired or wireless connections. Data communication may be performed within the same computer device, via an internet connection, or using short-range wireless communication technology. The type of communication protocol used for data communication is not limited.

[0026] The acquisition unit 21 acquires an image of the target product to determine its classification at a predetermined level within the tariff classification number. In other words, the acquisition unit 21 corresponds to the acquisition unit 11 in the classification support system 1. Typically, the acquisition unit 21 acquires an image of the target product captured using a camera. The image may be a still image or a series of images. In other words, the acquisition unit 21 may acquire an image of the target product by acquiring frames in a video. The acquisition unit 11 may include means for capturing an image of the target product. For example, the acquisition unit 11 may include a camera for capturing an image of the target product.

[0027] The image captured by the acquisition unit 21 is an image in which the characteristics of the product are represented to such an extent that a predetermined trained model can determine the classification of the target product in the tariff classification number at a predetermined level. If the target product is mass-producible, the image captured may be an image of the appearance of another mass-produced product that has the same appearance as the target product. Typically, the image captured is an image of the appearance of the target product. For example, if the target product is exported or imported in packaging made of predetermined packaging materials, the image captured is an image of the appearance of the target product in an unpackaged state. Furthermore, if the characteristics of the target product are located inside the product when determining the classification in the tariff classification number, the image captured by the acquisition unit 21 may be an image of the inside of the target product.

[0028] The acquisition unit 21 acquires at least a front view, side view, and top view of the target product as captured images of the target product. The acquisition unit 21 may also acquire a perspective view, rear view, and bottom view of the target product. In addition, the acquisition unit 21 may acquire an enlarged view of a part of the target product.

[0029] The acquisition unit 21 can acquire invoice information for the target product. Invoice information refers to the information written on the invoice. An invoice is also called a "purchase invoice." Invoice information includes, for example, the invoice number, date, product name, quantity, unit, unit price, amount, sender information, and recipient information. The acquisition unit 21 may acquire all of the above invoice information, or it may acquire only some of it. In this case, the acquisition unit 21 will acquire invoice information that includes at least the product name of the product.

[0030] The acquisition unit 21 can transmit the captured image of the target product and the invoice information of the product to the storage unit 22, the higher-level classification determination unit 23, the lower-level classification determination unit 24, and the classification assistance unit 25.

[0031] The storage unit 22 is composed of a storage device such as a hard disk or flash memory. The storage unit 22 stores the captured image of the product acquired by the acquisition unit 21 and the invoice information of the product. The storage unit 22 can store the captured image of the product and the invoice information of the product in association with each other.

[0032] The storage unit 22 can store images of target goods whose classification at a predetermined level has been determined. In other words, the storage unit 22 can store images of target goods whose classification at a predetermined level has been identified. Specifically, the storage unit 22 can store images of target goods whose "Chapter," "Heading," and "Subheading" in the tariff classification number have been identified. Here, such images are referred to as "master images." That is, the storage unit 22 can store master images. The master image may also be an image of a target product whose "Chapter" and "Heading" in the tariff classification number have been identified. Furthermore, the master image may be an image of the target product whose classification has been identified from the images acquired by the acquisition unit 21, or it may be an image not acquired by the acquisition unit 21. For example, the master image may be an image of the same product as the product targeted by the classification support system 2, and may have been acquired by another computer device. Hereafter, "imported image" refers to an image of a product whose classification at a predetermined hierarchical level targeted by the classification support system 2 has not yet been determined, and "master image" refers to an image of a product whose classification at a predetermined hierarchical level targeted by the classification support system 2 has already been determined.

[0033] The storage unit 22 can also store the classification of the goods depicted in the master image. Here, the classification of the goods includes the classification of a predetermined hierarchical level that the classification support system 2 intends to determine based on the goods' tariff classification number. If there is a higher hierarchical level above that level, the classification of the goods includes the classification of that higher hierarchical level. In other words, the storage unit 22 can store the master image in association with the classification of the higher hierarchical level and the classification of the predetermined hierarchical level of the goods. The storage unit 22 may further associate the goods' invoice information with this information.

[0034] Furthermore, the storage unit 22 can store text information describing the product. Text information is information generated based on the captured image of the product acquired by the acquisition unit 21. Specifically, the text information is information that describes the product shown in the captured image of the product acquired by the acquisition unit 21. The text information is typically text data that a user can understand, but is not limited to this. The text information may also be information expressed in a predetermined programming language, for example. The storage unit 22 can store text information that describes the product shown in an image in association with the image stored in the storage unit 22. The storage unit 22 may also store text information generated based on captured images that have not been acquired by the acquisition unit 21. For example, if the storage unit 22 is storing a master image received from another computer device, the storage unit 22 may receive and store text information for that master image from that computer device.

[0035] Furthermore, the memory unit 22 can store the invoice information of a product in association with an image of the product. The memory unit 22 can also store the feature vector of the image being stored. That is, the memory unit 22 can store a feature vector, composed of feature quantities extracted from the image of the product, in association with the image.

[0036] The higher-level classification determination unit 23 determines the classification to which the target product belongs from among multiple higher-level classifications in the tariff classification number. In other words, the higher-level classification determination unit 23 determines the classification to which the target product belongs from among multiple higher-level classifications. Here, the higher level refers to the "class" relative to the "heading," and the "class" or "heading" relative to the "sub-

[0037] The configuration of the higher-level classification determination unit 23 will now be described. Figure 4 is a block diagram showing the configuration of the higher-level classification determination unit 23. The higher-level classification determination unit 23 comprises an input unit 231 and an output unit 232. The input unit 231 inputs the captured image acquired by the acquisition unit 21 to a predetermined pre-trained model. Here, when the higher-level classification determination unit 23 determines the "category" of the target product, the pre-trained model input by the input unit 231 may be called the "category determination model." Similarly, when the higher-level classification determination unit 23 determines the "item," the pre-trained model may be called the "item determination model." Hereafter, the pre-trained model used in the higher-level classification determination unit 23 will be referred to as the "higher-level classification determination model."

[0038] A higher-level classification model, when given an image of a product as input, outputs classification information to which the product corresponding to the input image belongs at a higher level. This higher-level classification model can be generated using machine learning algorithms such as CNNs (neuronal neural networks) or decision trees. In the case of NN-based models, the higher-level classification model comprises an input layer, an output layer, and an intermediate layer. The input layer receives an image of the product as input. The output layer outputs classification information to which the product corresponding to the input image belongs at a higher level. The intermediate layer's parameters are learned using training data that takes images of products as input and outputs classification information to which the product belongs. This higher-level classification model takes an image of a product as input to the input layer, performs calculations in the intermediate layer, and outputs the classification information to which the product belongs from the output layer.

[0039] In the case of a decision tree-based approach, the machine learning algorithm may be gradient boosting, random forest, or other algorithms. In the decision tree-based approach, the higher-level classification model has a decision tree structure in which branching conditions have been learned using training data that takes images of products as input and outputs classification information to which the products belong. Such a higher-level classification model is configured to take captured images of products as input to the decision tree structure and output classification information to which the products belong through branching.

[0040] Here, the classification information to which the product belongs includes at least the classification number to which the product belongs. For example, when the higher-level classification determination unit 23 attempts to determine the "category" of a product, the classification information includes the number of the "category" to which the product specifically belongs. Here, the classification information may include information about multiple classifications. For example, the classification information may include the numbers of multiple "categories" to which the higher-level classification determination model determines to belong. The classification information may also include the confidence level of the classification determined by the higher-level classification determination model. For example, the classification information may include the numbers of multiple "categories" and the confidence level of the higher-level classification determination model for each "category". The confidence level of the classification may be expressed as a probability.

[0041] The input unit 231 can input not only the captured image acquired by the acquisition unit 21 but also a master image stored in the storage unit 22 to the higher-level classification determination model. In this case, the higher-level classification determination model outputs classification information to which the products in the captured image belong by comparing the similarity between the captured image and the master image. Specifically, the higher-level classification determination model determines the similarity between the feature vector of the captured image and the feature vector of the master image. Here, the similarity used by the higher-level classification determination model is, for example, cosine similarity. The higher-level classification determination model can then output classification information to which the products in the master image with high similarity belong. In this case, the higher-level classification determination model can calculate the confidence level of the classification based on the similarity with the master image.

[0042] The output unit 232 outputs classification information at a higher level corresponding to the captured image input by the input unit 231, based on the higher-level classification determination model. If the input unit 231 inputs a master image in addition to the captured image, the output unit 232 outputs classification information at a higher level corresponding to both the captured image and the master image. The output unit 232 can transmit classification information to the storage unit 22, the lower-level classification determination unit 24, and the display unit 26.

[0043] Returning to Figure 3, let's continue the explanation of the configuration of the classification support system 2. The sub-classification determination unit 24 determines the classification to which the target product belongs from among multiple classifications at a lower level relative to the higher level in the tariff classification number. In other words, the sub-classification determination unit 24 determines the classification at a predetermined level that the classification support system 1 determines for the target product. Specifically, the sub-classification determination unit 24 is configured to determine the "heading," "sub-

[0044] The configuration of the lower classification determination unit 24 will now be described. Figure 5 is a block diagram showing the configuration of the lower classification determination unit 24. The lower classification determination unit 24 comprises an input unit 241 and an output unit 242. The input unit 241 inputs the captured images acquired by the acquisition unit 21 to a predetermined trained model. The input unit 241 may also be called the "first input unit." That is, the input unit 241 corresponds to the input unit 12 in the classification support system 1. The trained model to which the input unit 241 inputs may also be called the "first trained model." Hereafter, the trained model used in the lower classification determination unit 24 will be referred to as the "first trained model."

[0045] The first trained model is a model generated for each of the multiple classifications included in a higher hierarchy that is higher than the lower hierarchy that the lower classification determination unit 24 is trying to classify. In other words, the first trained model is a model generated for each classification included in a higher hierarchy that is higher than the lower hierarchy in question. To put it another way, the first trained model is a model specialized for a particular classification included in that higher hierarchy.

[0046] The first pre-trained model may be generated for each classification in the hierarchy one level above the lower hierarchy to which the subclassification determination unit 24 is trying to determine the classification. For example, suppose that when the subclassification determination unit 24 tries to determine the "item" of a target product, the higher hierarchy, "category," is classified into "category A," "category B," and "category C." In this case, the first pre-trained model will be generated for each of "category A," "category B," and "category C." However, the first pre-trained model does not have to be generated for each classification in the hierarchy one level above the target lower hierarchy. For example, when the subclassification determination unit 24 tries to determine the "number" of a target product, the first pre-trained model used by the subclassification determination unit 24 may be a model generated specifically for the "category" to which the target product belongs.

[0047] The first pre-trained model, upon input of an image of a product, outputs classification information to which the product corresponding to the input image belongs at the lower level targeted by the lower classification determination unit 24. The first pre-trained model is generated using, for example, a machine learning algorithm based on a neural network (NN) such as a CNN or a decision tree machine learning algorithm. In the case of an NN-based model, the first pre-trained model comprises an input layer, an output layer, and an intermediate layer. The input layer takes the image of the product as input. The output layer outputs classification information to which the product corresponding to the input image belongs at the lower level. The parameters of the intermediate layer are learned using training data in which images of products are input and classification information to which products belong is output. This first pre-trained model takes the image of the product as input to the input layer, performs calculations in the intermediate layer, and outputs the classification information to which the product belongs from the output layer.

[0048] In the case of a decision tree-based approach, the machine learning algorithm may be gradient boosting, random forest, or other algorithms. In the decision tree-based approach, the first pre-trained model has a decision tree structure in which branching conditions have been learned using training data that takes images of products as input and outputs classification information to which the products belong at lower levels. Such a first pre-trained model is configured to take captured images of products as input to the decision tree structure and output classification information to which the products belong through branching.

[0049] The data used as training data by the first pre-trained model differs for each classification at the higher level relative to the lower level targeted by the lower classification determination unit 24. For example, suppose the higher level is "category," and "category" is classified into "category A," "category B," and "category C." In this case, the first pre-trained model corresponding to "category A" uses training data that takes images of products belonging to "category A" as input and outputs classification information to which the products belong at the target lower level. Similarly, the first pre-trained models corresponding to "category B" and "category C" use training data that takes images of products belonging to "category B" and "category C," respectively, as input.

[0050] The first trained model outputs classification information, including the classification number to which the product belongs. There may be one classification number or two or more. Here, the first trained model may determine the classification number to which the product belongs based on a predetermined threshold. That is, the first trained model can output classification information for one or more classifications that are above the predetermined threshold. The first trained model may also output classification information for classifications that are below the predetermined threshold.

[0051] Furthermore, the first trained model can output text information describing the product corresponding to the input image as classification information. This text information describes the product shown in the image, derived from the input image of the product. In other words, the first trained model can output information similar to the text information that the memory unit 22 can store as classification information.

[0052] Furthermore, the first trained model may include the confidence level of the classification by the first trained model in the classification information. Here, the confidence level of the classification may be expressed as a probability. If the first trained model outputs one or more classifications with a confidence level of above a predetermined threshold, it may output the confidence level of each classification in association with the one or two or more classifications. The first trained model may also output the confidence level of classifications below a predetermined threshold, in association with those classifications.

[0053] The first trained model may output text information as classification information if the confidence level of the classification is below a predetermined threshold. That is, if the confidence level of the classification is below a predetermined threshold for all classifications in the lower levels of the target, the first trained model may output text information as classification information. In other words, if there are classifications in the lower levels of the target where the confidence level is above a predetermined threshold, the first trained model does not need to output text information as classification information.

[0054] The input unit 241 inputs the captured image to the corresponding first trained model based on the classification information of the target product at a higher level output from the output unit 232 of the higher-level classification determination unit 23. In other words, the input unit 241 inputs the captured image to the first trained model generated for the higher-level classification to which the target product belongs. For example, if the higher level "Category" is classified into "Category A," "Category B," and "Category C," and the output unit 232 outputs classification information that the higher level of the target product is "Category A," the output unit 232 inputs the captured image to the first trained model corresponding to "Category A."

[0055] The input unit 241 can input not only the captured image acquired by the acquisition unit 21 but also a master image stored in the storage unit 22 to the first trained model. In this case, the first trained model outputs classification information to which the products in the captured image belong by determining the similarity between the captured image and the master image. Here, the similarity used by the first trained model is, for example, cosine similarity. The first trained model can then output classification information to which the products in the master image with high similarity belong. In this case, the first trained model can calculate the confidence level of the classification based on the similarity with the master image.

[0056] The output unit 242 outputs classification information at a lower level corresponding to the captured image input by the input unit 241, based on the first classification determination model. The output unit 242 may also be referred to as the "first output unit." That is, the output unit 242 corresponds to the output unit 13 in the classification support system 1. Here, if the input unit 241 inputs a master image in addition to the captured image, the output unit 242 outputs classification information at a lower level corresponding to both the captured image and the master image. The output unit 242 can transmit classification information to the storage unit 22, the classification assistance unit 25, and the display unit 26.

[0057] The classification information output by the output unit 242 is the classification information output by the first trained model. That is, the output unit 242 can output classification information that includes the classification number to which the product belongs and the confidence level of the classification by the first trained model. In addition, the output unit 242 can output text information corresponding to the captured image input by the input unit 241. Here, the output unit 242 can output the text information as classification information if the confidence level of the classification by the first trained model is below a predetermined threshold.

[0058] Returning to Figure 3, let's continue explaining the configuration of the classification support system 2. The classification assistance unit 25 determines the classification to which the product belongs at a predetermined hierarchy corresponding to the text information describing the product. Here, the predetermined hierarchy is the hierarchy targeted by the subordinate classification determination unit 24. In other words, the classification assistance unit 25 determines the classification information at the lower hierarchy targeted by the subordinate classification determination unit 24. The classification assistance unit 25 is configured to assist the classification determination by the subordinate classification determination unit 24. Specifically, the classification assistance unit 25 is configured to determine the "heading" or "sub-heading" or both in the tariff classification number. While the subordinate classification determination unit 24 uses an image classification model, the classification assistance unit 25 uses a text classification model.

[0059] Furthermore, the classification auxiliary unit 25 may determine the classification information at a higher level for the lower level targeted by the lower classification determination unit 24. In other words, the classification auxiliary unit 25 may determine the classification information at a higher level targeted by the higher classification determination unit 23. Specifically, the classification auxiliary unit 25 may determine the "class" in the tariff classification number. That is, the classification auxiliary unit 25 may determine any tariff classification number for the target product.

[0060] The configuration of the classification assistance unit 25 will now be described. Figure 6 is a block diagram showing the configuration of the classification assistance unit 25. The classification assistance unit 25 comprises an input unit 251 and an output unit 252. The input unit 251 receives text information describing a product and inputs the text information output from the output unit 242 to a trained model that outputs classification information at a predetermined level corresponding to the text information. The input unit 251 may also be called the "second input unit." The trained model to which the input unit 251 receives input may also be called the "second trained model." Hereafter, the trained model used in the classification assistance unit 25 will be referred to as the "second trained model." The second trained model may also output an arbitrary tariff classification number for a product when text information describing the product is input.

[0061] The second pre-trained model outputs classification information for a predetermined hierarchy corresponding to text information when text information describing a target product is input to the classification assistance unit 25. The second pre-trained model is generated using machine learning algorithms such as a Support Vector Machine (SVM), decision tree, or CNN. In the case of an SVM-based model, the classification decision boundary of the second pre-trained model is learned using training data in which text information describing a product is input and classification information to which the product belongs is output. As a result, this second pre-trained model can take text information describing a product as input and output classification information to which the product belongs. The explanation of the second pre-trained model using decision tree and CNN machine learning algorithms is omitted. Furthermore, the classification information output by the second pre-trained model is the same as the classification information output by the first pre-trained model used by the lower-level classification decision unit 24.

[0062] The input unit 251 may also receive invoice information for the product acquired by the acquisition unit 21, in addition to text information describing the product. If invoice information is also input, the second trained model outputs classification information based on the invoice information and the text information describing the product. In other words, the second trained model outputs classification information from the content of the text information that is included in the invoice information. In this case, the second trained model learns using training data that takes invoice information as input during the learning phase.

[0063] The output unit 252 outputs classification information at a predetermined level corresponding to the text information input by the input unit 251 from the second trained model. The output unit 252 may also be referred to as the "second output unit". If the second trained model outputs an arbitrary tariff classification number for a product when text information describing the product is input, the output unit 252 outputs the tariff classification number.

[0064] If the input unit 251 inputs invoice information in addition to text information, the output unit 252 outputs classification information at a predetermined hierarchy corresponding to the text information and the invoice information. If the second trained model outputs an arbitrary tariff classification number for a product when text information describing the product is input, the output unit 252 outputs the tariff classification number corresponding to the text information and the invoice information. The output unit 252 can transmit the classification information to the storage unit 22 and the display unit 26.

[0065] Returning to Figure 3, we continue the explanation of the configuration of the classification support system 2. The display unit 26 is a display device including a liquid crystal panel or electroluminescence. The display unit 26 may be mounted on a user terminal owned by the user. Here, the user terminal is, for example, a notebook PC (Personal Computer), a smartphone, or a tablet terminal. The display unit 26 can display classification information output by the output unit 232 of the higher-level classification determination unit 23, the output unit 242 of the lower-level classification determination unit 24, and the output unit 252 of the classification assistance unit 25 to the user. The display unit 26 can also display the captured image of the target product acquired by the acquisition unit 21. The display unit 26 can display this information in any format. For example, the display unit 26 can display the captured image and classification information acquired by the acquisition unit 21 in association with each other.

[0066] Next, the flow of the classification support method by the classification support system 2 will be explained. Figure 6 is a flowchart showing an example of the flow of the classification support method by the classification support system 2. First, the acquisition unit 21 acquires an image of the target product (S201). Here, the acquisition unit 21 can also acquire invoice information for the target product. Next, the input unit 231 of the higher-level classification determination unit 23 inputs the image of the target product acquired by the acquisition unit 21 to the higher-level classification determination model (S202). Here, the input unit 231 may also input a master image stored in the storage unit 22. In this case, the higher-level classification determination model determines the similarity between the image and the master image. After that, the output unit 232 outputs classification information at a higher level corresponding to the image input by the input unit 231 from the higher-level classification determination model (S203). Here, if the input unit 231 inputs a master image in addition to the image, the output unit 232 outputs classification information at a higher level corresponding to the image and the master image.

[0067] Subsequently, the input unit 241 of the lower classification determination unit 24 inputs the captured images acquired by the acquisition unit 21 to the first trained model (S204). Here, the first trained model is a model generated for classifications determined by the upper classification determination unit 23, which are located above the lower hierarchy targeted by the lower classification determination unit 24. In other words, the input unit 241 inputs the captured images acquired by the acquisition unit 21 to the first trained model, which was generated for classifications determined by the upper classification determination unit 23. In addition to the captured images taken by the acquisition unit 21, the input unit 241 may also input master images stored by the classification support system 2 to the first trained model.

[0068] Subsequently, the first trained model determines whether the confidence level of the classification at the lower level is below a predetermined threshold (S205). If the result in step S205 is YES, the output unit 242 outputs text information as classification information from the first trained model (S206). Here, the output unit 242 may also output the classification number and the confidence level of the classifications that are below the threshold. On the other hand, if the result in step S205 is NO, the output unit 242 outputs the classification number and the confidence level of the classifications that are equal to or greater than the predetermined threshold as classification information (S207). Here, the output unit 242 may further output the classification number and the confidence level of the classifications that are below the threshold.

[0069] In step S206, after the output unit 242 outputs text information as classification information, the input unit 251 of the classification assistance unit 25 inputs the text information output by the output unit 242 into the second trained model (S208). Here, the second trained model is a model for determining the same classification as the lower classification that the lower classification determination unit 24 is targeting for determination. In addition to the text information, the input unit 251 may also input invoice information acquired by the acquisition unit 21. Subsequently, the output unit 252 outputs classification information from the second trained model that corresponds to the text information input by the input unit 251 (S209). Here, if the input unit 251 inputs invoice information in addition to text information, the output unit 252 outputs classification information at a predetermined hierarchy corresponding to the text information and the invoice information.

[0070] Subsequently, the acquisition unit 21, the output unit 232 of the higher-level classification determination unit 23, the output unit 242 of the lower-level classification determination unit 24, and the output unit 252 of the classification assistance unit 25 store the captured image and classification information in the storage unit 22 (S210). Also, after the processing in step S207, the acquisition unit 21, the output unit 232 of the higher-level classification determination unit 23, and the output unit 242 of the lower-level classification determination unit 24 similarly store the captured image and classification information in the storage unit 22. Then, the display unit 26 displays the classification information to the user (S211). Here, steps S210 and S211 may be performed simultaneously, or step S211 may be performed before S210.

[0071] In this way, the classification support system 2 can assist in determining the tariff classification number. Specifically, the classification support system 2 can determine the higher-level classification in the tariff classification number based on the captured image of the target product, using a trained model (higher-level classification determination model). Then, the classification support system 2 can determine the lower-level classification for that higher-level classification by using a trained model (first trained model) based on the classification information of that higher level. As a result, the classification support system 2 can determine the classification from the higher level to the lower level in the tariff classification number.

[0072] For example, related technologies can determine tariff classification numbers using Large Language Models (LLMs). In such technologies, users create text information to input into the LLM using invoice information associated with the target product for which they want to determine the tariff classification number. The LLM determines the tariff classification number of the target product based on the text information created by the user. However, the text information created by users based on invoice information may differ from user to user. Therefore, the accuracy of determining tariff classification numbers based on such text information may vary. Consequently, the process of creating text information using invoice information may become dependent on individual users.

[0073] The classification support system 2 can determine the tariff classification number based on the captured image of the target product. In other words, the classification support system 2 can determine the tariff classification number using information based on the appearance of the target product. Therefore, the classification support system 2 can suppress variations in the accuracy of determining the tariff classification number. Furthermore, the classification support system 2 can output text information describing the product corresponding to the captured image of the target product. In other words, the classification support system 2 can create the text information using a trained model (first trained model) rather than the user. Because the classification support system 2 can create user-independent text information, it can suppress the personalization of text information creation by users.

[0074] Furthermore, the classification support system 2 can output text information for a product if the confidence level of the classification at a predetermined hierarchical level based on the captured image of the product is below a predetermined threshold. In other words, the classification support system 2 can generate text information when it is not possible to determine an appropriate classification based on the captured image. Therefore, users such as customs brokers can use the generated text information to determine the classification in such cases.

[0075] Furthermore, the classification support system 2 can output classification information at a predetermined hierarchy using text information. Therefore, the classification support system 2 can determine the classification of a product not only using the captured image of the product, but also using text information that describes the product. In particular, the classification support system 2 can determine the classification of a product using the text information generated when it is not possible to determine an appropriate classification based on the captured image. Therefore, the classification support system 2 can determine the classification of a product even when it is not possible to determine an appropriate classification based on the captured image.

[0076] Furthermore, the classification support system 2 can determine the classification of a product based on the information contained in the invoice information of the target product among the generated text information. In other words, the classification support system 2 can determine the classification of a target product by taking the invoice information into consideration.

[0077] Furthermore, the classification support system 2 can determine the classification of a target product by determining the similarity between the captured image of the target product to be classified and the master image of a product whose classification has already been determined. In other words, when the classification support system 2 attempts to determine the classification of a target product using a master image, it can automatically perform comparisons between images. Therefore, the classification support system 2 can improve user convenience.

[0078] Furthermore, when the classification support system 2 attempts to determine the classification to which the target product belongs at a predetermined level of the tariff classification number, it can use a pre-trained model (first pre-trained model) that has been generated for each classification in the level one level above the predetermined level. In addition, the classification support system 2 can acquire at least a front view, side view, and top view of the target product as captured images of the product. As a result, the classification support system 2 can determine the classification with high accuracy.

[0079] (Example hardware configuration) Figure 7 shows an example of the hardware configuration of the classification support system 3. In Figure 7, the classification support system 3 has a processor 31 and a memory 32. The processor 31 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 31 may include multiple processors. The memory 32 is composed of a combination of volatile memory and non-volatile memory. The memory 32 may include storage located away from the processor 31. In this case, the processor 31 may access the memory 32 via an I / O (Input / Output) interface, which is not shown.

[0080] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Alternatively, the program may be provided to the computer using various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels. Computers include various information processing devices such as PCs, servers, CPUs, MPUs, FPGAs (Field Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits).

[0081] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0082] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0083] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) A classification support system that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, An acquisition unit that acquires an image of the target product, A first input unit inputs the captured image acquired by the acquisition unit to a first trained model that, when an image of a product is input, outputs classification information to which the product corresponding to the input image belongs at a predetermined hierarchy, The system includes a first output unit that outputs classification information at a predetermined level corresponding to the captured image input by the first input unit from the first trained model, The first trained model is generated for each of the multiple classifications included in a higher hierarchy that is located above the predetermined hierarchy, The first input unit inputs the captured image to the first trained model generated for the classification in the higher hierarchy to which the target product belongs. Classification support system. (Note 2) The first trained model outputs text information describing the product corresponding to the input image as classification information. The first output unit outputs the text information as classification information in the predetermined hierarchy corresponding to the captured image input by the first input unit. The classification support system described in Appendix 1. (Note 3) The first output unit outputs the text information as classification information when the confidence level of the classification in the predetermined hierarchy corresponding to the captured image is less than a predetermined threshold. The classification support system described in Appendix 2. (Note 4) A second input unit inputs the text information output from the first output unit into a second trained model that outputs a tariff classification number for a product when text information describing the product is input. The system further comprises a second output unit that outputs a tariff classification number corresponding to the text information input by the second input unit from the second trained model, The classification support system described in Appendix 2 or 3. (Note 5) The acquisition unit acquires the invoice information of the target product, The second trained model outputs a tariff classification number for a product when text information describing the product, which is included in the invoice information, is entered. The classification support system described in Appendix 4. (Note 6) The system further includes a storage unit that stores a master image of a target product whose classification at a predetermined hierarchical level has been determined, and associates the higher-level classification of the target product with the classification at the predetermined hierarchical level. The first input unit further inputs the master image to a first trained model that outputs classification information of a predetermined hierarchy for the target product by determining the similarity between the captured image acquired by the acquisition unit and the master image belonging to a higher-level classification of the target product shown in the captured image. The first output unit outputs classification information of a predetermined hierarchy corresponding to the captured image and the master image input by the first input unit. A classification support system described in any one of the items 1 to 5 in the appendix. (Note 7) The first trained model is generated for each classification that is included in the hierarchy one level above the predetermined hierarchy for which the classification is to be determined. A classification support system as described in any one of the items 1 to 6 of the appendix. (Note 8) The acquisition unit acquires at least a front view, a side view, and a top view of the target product as captured images of the target product. A classification support system as described in any one of the items 1 through 7 of the appendix. (Note 9) Computers A classification support method for determining the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, Acquire an image of the aforementioned product, Among multiple classifications located in a higher hierarchy above the predetermined hierarchy, a first trained model is generated for the classification in the higher hierarchy to which the target product belongs, and when an image of the product is input, it outputs classification information to which the product corresponding to the input image belongs in the predetermined hierarchy. The acquired image is then input to this first trained model. From the first trained model, classification information at the predetermined hierarchy corresponding to the input captured image is output. Classification support method. (Note 10) A program that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, The steps include: acquiring an image of the target product; The steps include inputting the acquired image to a first trained model which generates information for the classification in the higher hierarchy to which the target product belongs, among multiple classifications located in a higher hierarchy above the predetermined hierarchy, and which outputs classification information to which the product corresponding to the input image belongs in the predetermined hierarchy when an image of the product is input, The steps include outputting classification information at a predetermined hierarchy corresponding to the input captured image from the first trained model, A program that causes a computer to execute something. (Note 11) Training data is obtained that includes an image of the product and classification information to which the product belongs at a predetermined level in a hierarchical tariff classification number. Based on the aforementioned training data, a learning model is generated that outputs classification information to which the product belongs when an image of the product is input. Method for generating a learning model. (Note 12) An input layer into which the captured image of the product is input, An output layer that outputs classification information to which the product belongs at a predetermined level in a tariff classification number having a hierarchical structure, The system includes an intermediate layer whose parameters have been learned using training data in which captured images of the aforementioned product are input and classification information to which the aforementioned product belongs is output. The system takes an image of the product as input, performs calculations in the intermediate layer, and outputs classification information to which the product belongs from the output layer. A pre-trained model for use with computers.

[0084] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 8 that are dependent on Appendice 1 may also be dependent on Appendices 9 to 12 in the same manner as those described in Appendices 2 to 8. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software.

[0085] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its intent. For example, this disclosure is not limited to tariff classification numbers. This disclosure is applicable to other numbers and codes that have a hierarchical structure and are used to classify goods. [Explanation of symbols]

[0086] 1. Classification support system 2. Classification Support System 3. Classification Support System 11 Acquisition Department 12 Input section 13 Output section 21 Acquisition Department 22 Memory section 23 Upper classification determination section 24 Lower classification determination section 25 Classification auxiliary part 26 Display section 31 processors 32 memory 231 Input section 232 Output section 241 Input section 242 Output section 251 Input section 252 Output section

Claims

1. A classification support system that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, An acquisition unit that acquires an image of the target product, A first input unit inputs the captured image acquired by the acquisition unit to a first trained model that, when an image of a product is input, outputs classification information to which the product corresponding to the input image belongs at a predetermined hierarchy, The system includes a first output unit that outputs classification information at a predetermined level corresponding to the captured image input by the first input unit from the first trained model, The first trained model is generated for each of the multiple classifications included in the higher hierarchy that is located above the predetermined hierarchy, The first input unit inputs the captured image to the first trained model generated for the classification in the higher hierarchy to which the target product belongs. Classification support system.

2. The first trained model outputs text information describing the product corresponding to the input image as classification information. The first output unit outputs the text information as classification information in the predetermined hierarchy corresponding to the captured image input by the first input unit. The classification support system according to claim 1.

3. The first output unit outputs the text information as classification information when the confidence level of the classification in the predetermined hierarchy corresponding to the captured image is less than a predetermined threshold. The classification support system according to claim 2.

4. A second input unit inputs the text information output from the first output unit into a second trained model that outputs a tariff classification number for a product when text information describing the product is input. The system further comprises a second output unit that outputs a customs classification number corresponding to the text information input by the second input unit from the second trained model, The classification support system according to claim 2 or 3.

5. The acquisition unit acquires the invoice information of the target product, The second trained model outputs a customs classification number for a product when text information describing the product, which is included in the invoice information, is entered. The classification support system according to claim 4.

6. The system further includes a storage unit that stores a master image of a target product whose classification at a predetermined hierarchical level has been determined, and associates the higher-level classification of the target product with the classification at the predetermined hierarchical level. The first input unit further inputs the master image to a first trained model that outputs classification information of a predetermined hierarchy for the target product by determining the similarity between the captured image acquired by the acquisition unit and the master image belonging to a higher-level classification of the target product shown in the captured image. The first output unit outputs classification information of a predetermined hierarchy corresponding to the captured image and the master image input by the first input unit. The classification support system according to claim 1.

7. The first trained model is generated for each classification that is included in the hierarchy one level above the predetermined hierarchy for which the classification is to be determined. The classification support system according to claim 1.

8. The acquisition unit acquires at least a front view, a side view, and a top view of the target product as captured images of the target product. The classification support system according to claim 1.

9. Computers A classification support method for determining the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, Acquire an image of the aforementioned product, Among multiple classifications located in a higher hierarchy above the predetermined hierarchy, a first trained model is generated for the classification in the higher hierarchy to which the target product belongs, and when an image of the product is input, it outputs classification information to which the product corresponding to the input image belongs in the predetermined hierarchy. The acquired image is then input to this first trained model. From the first trained model, classification information at the predetermined hierarchy corresponding to the input captured image is output. Classification support method.

10. A program that determines the classification to which a target product belongs from among multiple classifications at a predetermined hierarchical level in a tariff classification number, The steps include: acquiring an image of the target product; The steps include inputting the acquired image to a first trained model which generates information for the classification in the higher hierarchy to which the target product belongs, among multiple classifications located in a higher hierarchy above the predetermined hierarchy, and which outputs classification information to which the product corresponding to the input image belongs in the predetermined hierarchy when an image of the product is input, The first step of outputting classification information at a predetermined hierarchy corresponding to the input captured image from the first trained model, A program that causes a computer to execute something.

Images