Appraisal support device and appraisal support system

Abstract

We can appraise products with less time and effort than before. [Solution] Server device 2 receives at least one input image representing one product 6 from terminal device 1. Server device 2 converts the input image into a first feature vector using a product model trained to output different feature vectors when images of different products are input. Server device 2 determines at least one second feature vector similar to the first feature vector from among a plurality of second feature vectors converted using the product model from a plurality of product images corresponding to product information stored in storage device 23. Server device 2 reads the product information corresponding to the second feature vector similar to the first feature vector from storage device 23 as candidate product information corresponding to product 6 in the input image. Server device 2 transmits the candidate product information to terminal device 3.

Description

Technical Field

[0001] The present disclosure relates to an appraisal support device and an appraisal support system.

Background Art

[0002] When appraising a product for used product trading etc., in-depth knowledge about the product is required, and even for those with specialized knowledge, appraising is time-consuming. Therefore, it is required to support the appraisal of products.

[0003] Patent Document 1 discloses a product appraisal method for appraising items such as artworks and antiques on a computer network.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The method described in Patent Document 1 requires inputting attribute data of the item to be appraised (for example, artwork code, classification code, photographic image data, name (title), artist name, size, date, etc.) and specified data that defines the evaluation base. Inputting this data is time-consuming and results in a significant loss of time when appraising a large number of items. Therefore, there is a need to support item appraisal so that items can be appraised with less time and effort than before.

[0007] The purpose of this disclosure is to provide an appraisal support device and an appraisal support system that can appraise products with less time and effort than before. [Means for solving the problem]

[0008] An appraisal support device according to a first aspect of this disclosure comprises a communication device connected to first and second terminal devices via a communication line, a storage device for storing product information relating to a plurality of products, and a processor. The processor receives at least one input image representing a single product from the first terminal device. The processor converts the input image into a first feature vector using a product model trained to output different feature vectors when images of different products are input. The processor determines at least one second feature vector similar to the first feature vector from among a plurality of second feature vectors converted using the product model from images of a plurality of products corresponding to the product information stored in the storage device. The processor reads product information corresponding to the second feature vector similar to the first feature vector from the storage device as candidate product information corresponding to the product in the input image. The processor transmits the candidate product information to the second terminal device.

[0009] According to the appraisal support device of the second aspect of this disclosure, in the appraisal support device of the first aspect, the product model includes a convolutional neural network and a set transformer.

[0010] According to the assessment support device of the third aspect of this disclosure, in the assessment support device of the first or second aspect, the processor determines at least one second feature vector similar to the first feature vector by performing a k-nearest neighbor search in a vector space including the first and second feature vectors.

[0011] According to the appraisal support device of the fourth aspect of this disclosure, in the appraisal support device of one of the first to third aspects, the processor trains the product model by inputting images associated with product information having a first granularity into the product model. The processor generates the second feature vector by inputting images associated with product information having a second granularity, which is finer than the first granularity, into the product model.

[0012] According to the appraisal support device of the fifth aspect of this disclosure, in the appraisal support device of one of the first to fourth aspects, the processor receives appraisal information of the product in the input image from the second terminal device and transmits the appraisal information to the first terminal device.

[0013] The assessment support system according to the sixth aspect of this disclosure includes the first terminal device, the second terminal device, and an assessment support device according to one of the first to fifth aspects,

[0014] A seventh aspect of the present disclosure of an appraisal support system includes a first terminal device, a second terminal device, and an appraisal support device connected to the first and second terminal devices via a communication line. The first terminal device includes a camera, a first display device, and a first processor. The second terminal device includes a second display device, an input device, and a second processor. The appraisal support device includes a storage device for storing product information relating to a plurality of products and a third processor. The first processor generates at least one input image showing one product using the camera and transmits the input image to the appraisal support device. The third processor receives the input image from the first terminal device, recognizes the input image, reads product information corresponding to the product in the input image from the storage device as candidate product information, and transmits the candidate product information to the second terminal device. The second processor receives the candidate product information from the appraisal support device, displays the candidate product information on the second display device, obtains appraisal information for the product in the input image via the input device, and transmits the appraisal information to the appraisal support device. The third processor receives the appraisal information from the second terminal device and transmits the appraisal information to the first terminal device. The first processor receives the appraisal information from the appraisal support device and displays the appraisal information on the first display device. The candidate product information includes the product's image, name, and past transaction history.

[0015] According to the appraisal support system of the eighth aspect of this disclosure, in the appraisal support system of the seventh aspect, the second display device displays the input image and the image of the product included in the candidate product information in a comparable manner.

[0016] According to the appraisal support system of the ninth aspect of this disclosure, in the appraisal support system of the seventh or eighth aspect, the third processor recognizes a plurality of input images representing a single product, and reads from the storage device at least one set of product information corresponding to one of the products in the input images as the candidate product information. [Effects of the Invention]

[0017] An appraisal support device and an appraisal support system according to an aspect of the present disclosure can appraise a product with less time and effort than in the past.

Brief Description of the Drawings

[0018] [Figure 1] It is a schematic diagram showing the configuration of the appraisal support system 100 according to the embodiment. [Figure 2] It is a block diagram showing the configuration of the terminal device 1 in FIG. 1. [Figure 3] It is a block diagram showing the configuration of the server device 2 in FIG. 1. [Figure 4] It is a block diagram showing the configuration of the terminal device 3 in FIG. 1. [Figure 5] It is a sequence diagram showing the operation of the appraisal support system 100 in FIG. 1. [Figure 6] It is a diagram for explaining the extraction of the feature vector in step S2 in FIG. 5. [Figure 7] It is a diagram for explaining the search of the feature vector in step S3 in FIG. 5. [Figure 8] It is a diagram showing the candidate product information displayed on the terminal device 3 in step S5 in FIG. 5. [Figure 9] It is a diagram showing the appraisal information displayed on the terminal device 1 in step S7 in FIG. 5. [Figure 10] It is a diagram for explaining the first loss function used when training the product model 40 in FIG. 6. [Figure 11] It is a diagram for explaining the second loss function used when training the product model 40 in FIG. 6.

Modes for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The same reference numerals denote the same components.

[0020] [Configuration of the Embodiment] Figure 1 is a schematic diagram showing the configuration of the appraisal support system 100 according to an embodiment. The appraisal support system 100 includes a terminal device 1, a server device 2, a terminal device 3, a base station device 4, and a communication line 5. The appraisal support system 100 assists in the appraisal of goods 6 for the buying and selling of used goods.

[0021] Terminal device 1 is an electronic device equipped with a camera 17, such as a mobile phone. Terminal device 1 is communicatively connected to server device 2 via base station device 4 and communication line 5. Terminal device 1 is carried by a sales representative who visits the owner of product 6.

[0022] Server device 2 stores product information for multiple products and assists in the appraisal of product 6. Server device 2 is an example of an appraisal support device.

[0023] Terminal device 3 is connected to server device 2 via communication line 5 for communication. Terminal device 3 is operated by an appraiser who assesses product 6.

[0024] Communication line 5 is, for example, a local area network (LAN), the internet, a mobile phone network, or a combination thereof.

[0025] Figure 2 is a block diagram showing the configuration of terminal device 1 in Figure 1. Terminal device 1 comprises a bus 10, a processor 11, memory 12, storage device 13, communication device 14, input device 15, display device 16, and camera 17. The processor 11 controls the operation of the entire terminal device 1. Memory 12 temporarily stores programs and data necessary for the operation of terminal device 1. Storage device 13 is a non-volatile storage medium that stores programs necessary for the operation of terminal device 1. The communication device 14 is communicably connected to server device 2 via base station device 4 and communication line 5. Input device 15 receives user input to control the operation of terminal device 1. Input device 15 includes, for example, switches and a touch panel. Display device 16 displays information related to the status of terminal device 1 and also displays appraisal information of product 6 determined by the appraiser. Camera 17 photographs product 6. Terminal device 1 transmits the photographed image of product 6 to server device 2 via base station device 4 and communication line 5. The processor 11, memory 12, storage device 13, communication device 14, input device 15, display device 16, and camera 17 are connected to each other via the bus 10.

[0026] Figure 3 is a block diagram showing the configuration of the server device 2 in Figure 1. The server device 2 comprises a bus 20, a processor 21, memory 22, storage device 23, and a communication device 24. The processor 21 controls the operation of the entire server device 2 and also assists the appraiser in appraising the product 6. The memory 22 temporarily stores the programs and data necessary for the operation of the server device 2. The storage device 23 is a non-volatile storage medium that stores the programs and data necessary for the operation of the server device 2. The data necessary for the operation of the server device 2 includes, for example, product information for multiple products. The communication device 24 is connected to terminal devices 1 and 3 via a communication line 5. The processor 21, memory 22, storage device 23, and communication device 24 are connected to each other via the bus 20.

[0027] The product information stored in the storage device 23 includes, for example, the following items:

[0028] Identification number, image, type, name (product name, brand name, line name, etc.), model number, material, color, accessories, gender (men's, women's), size (S, M, L), dimensions (length, width, height), capacity, grade, alcohol content, condition (good, dirty, scratched, junk, etc.), free-form description of features, and / or transaction history (past purchase price, past selling price).

[0029] Each item in the product information is a label associated with the product. Multiple products are classified into multiple classes based on these labels.

[0030] Product information is managed by administrators with product knowledge, such as those in the department responsible for product assessment.

[0031] Transaction history may be managed separately for each item's condition (good, dirty, scratched, junk, etc.).

[0032] Figure 4 is a block diagram showing the configuration of the terminal device 3 in Figure 1. The terminal device 3 comprises a bus 30, a processor 31, a memory 32, a storage device 33, a communication device 34, an input device 35, and a display device 36. The processor 31 controls the operation of the entire terminal device 3. The memory 32 temporarily stores programs and data necessary for the operation of the terminal device 3. The storage device 33 is a non-volatile storage medium that stores programs necessary for the operation of the terminal device 3. The communication device 34 is communicably connected to the server device 2 via a communication line 5. The input device 35 receives user input that controls the operation of the terminal device 3. The input device 35 includes, for example, a keyboard and a pointing device. The display device 36 displays information related to the status of the terminal device 3 and also displays information related to the goods 6 being assessed. The processor 31, memory 32, storage device 33, communication device 34, input device 35, and display device 36 are connected to each other via the bus 30.

[0033] [Operation of the Embodiment] Figure 5 is a sequence diagram showing the operation of the assessment support system 100 in Figure 1.

[0034] In step S1, the processor 11 of the terminal device 1 uses the camera 47 to photograph a single product 6 and generates one or more input images representing the product 6. If the front view of the product 6 contains sufficient information to identify the product 6, only one input image may be used. If images including views other than the front of the product 6 are necessary to identify the product 6, multiple input images representing the product 6 may be used. The processor 11 transmits the input images to the server device 2.

[0035] The processor 21 of server device 2 receives an input image from terminal device 1. Next, the processor 21 recognizes the input image by executing steps S2 to S4 and reads product information corresponding to the product in the input image from storage device 23 as candidate product information.

[0036] In step S2, the processor 21 of the server device 2 uses a pre-trained product model to convert one or more input images into a single feature vector (the first feature vector). The product model is trained to output different feature vectors when images of different products are input.

[0037] In step S3, the processor 21 of the server device 2 searches for a feature vector similar to the first feature vector in the pre-generated feature vector space (feature space). The feature space includes multiple feature vectors (second feature vectors) that are each transformed using a product model from multiple product images corresponding to product information stored in the storage device 23. The processor 21 determines at least one second feature vector from the multiple second feature vectors that is similar to the first feature vector. Since the second feature vector is generated from product images corresponding to product information stored in the storage device 23, it is associated with product information, i.e., the product label.

[0038] In step S4, the processor 21 of the server device 2 reads product information corresponding to a second feature vector similar to the first feature vector from the storage device 23 as candidate product information corresponding to product 6 in the input image. In other words, the processor 21 reads product information from the storage device 23 that matches or is similar to the label of the product estimated to correspond to product 6 in the input image as candidate product information. The candidate product information includes, for example, the product's image, name, and past transaction history. The processor 21 transmits the candidate product information to the terminal device 3.

[0039] The processor 31 of terminal device 3 receives candidate product information from server device 2.

[0040] In step S5, the processor 31 of the terminal device 3 displays candidate product information on the display device 36. Based on the candidate product information, the appraiser determines the appraisal information for product 6, such as the purchase price of product 6.

[0041] In step S6, the processor 31 of the terminal device 3 obtains appraisal information for the product 6 in the input image via the input device. The processor 31 transmits the appraisal information to the server device 2.

[0042] The processor 21 of server device 2 receives assessment information from terminal device 3 and transmits the assessment information to terminal device 1.

[0043] The processor 11 of terminal device 1 receives assessment information from server device 2.

[0044] In step S7, the processor 11 of the terminal device 1 displays the assessment information on the display device 16.

[0045] As shown in Figure 5, when the appraisal support system 100 is in operation, the sales representative (user of terminal device 1) only needs to take a picture of the product 6 and send the image to the server device 2, and the appraiser (user of terminal device 3) only needs to check the product information provided by the server device 2. The appraisal support system 100 eliminates or reduces the need for sales representatives and appraisers to manually input product information and can support the appraisal of product 6 by the appraiser.

[0046] The processor 21 of the server device 2 may recognize multiple input images representing a single product and read at least one set of product information corresponding to one of the products in the input images from the storage device 23 as candidate product information.

[0047] The processor 21 of the server device 2 may transmit the input image and the label of the product estimated to correspond to product 6 in the input image to the terminal device 3, along with the candidate product information. In this case, the processor 31 of the terminal device 3 displays the input image and label along with the candidate product information on the display device 36.

[0048] The processor 21 of server device 2 identifies the product 6 in the input image in two steps S2 and S3. In step S2, the processor 21 uses a pre-trained product model to convert one or more input images into a single feature vector (feature extraction). In step S3, the processor 21 searches the feature space for feature vectors similar to the feature vector from step S2 (neighborhood search). By performing a neighborhood search in the feature space using the feature vector as a query, the class of product 6 in the input image is determined.

[0049] Figure 6 illustrates the extraction of feature vectors in step S2 of Figure 5. The server device 2 has a pre-trained product model 40. The product model 40 includes a convolutional neural network (CNN) 41 and a set transformer 42. In the embodiment of Figure 6, the product model 40 includes a cascaded connection of the convolutional neural network 41 and the set transformer 42.

[0050] The convolutional neural network 41 transforms the input images 43-1 to 43-4 into image features x1 to x4, respectively. The features x1 to x4 are then sent to the set transformer 42.

[0051] The set transformer 42 converts a predetermined number of feature vectors x1 to x4 into a single feature vector. The set transformer is described, for example, in Non-Patent Literature 1. The set transformer 42 comprises an encoder 42a and a decoder 42b. The encoder 42a extracts data z relating to the interaction of each pair of input feature vectors. The decoder 42b combines the data z output from the encoder 42a to generate a single feature vector v. The set transformer 42 dynamically determines the weights of the feature vectors x1 to x4 from the feature vectors x1 to x4 themselves. This can be considered as the set transformer 42 dynamically selecting the feature vectors corresponding to important input images.

[0052] The feature vector v output from the set transformer 42 is invariant with respect to the order of the features x1 to x4 input to the set transformer 42.

[0053] The set transformer 42 requires a predetermined number of feature quantities as input, but the number of input images generated by a sales representative when photographing product 6 is not constant. In the example in Figure 6, only input images 43-1 to 43-3 show product 6, and input image 43-4 is blank or padding data. In this case, the product model 40 sends the feature quantities (x1×1, x2×1, x3×1, x4×0) of the masked image using mask data (1,1,1,0) to the subsequent set transformer 42. This allows variable-length input image data to be treated as a fixed-length tensor.

[0054] Product model 40 has the following characteristics by including a combination of a convolutional neural network 41 and a set transformer 42. • Converts one or more input images representing a single product into a single feature vector. • If multiple input images representing a single product are provided, the value of the feature vector remains unchanged even if the order of the input images is changed. • If one set of input images and another set of input images represent the same product 6, they will be converted into the same or similar feature vectors, regardless of the number of input images in each set.

[0055] Figure 7 illustrates the search for feature vectors in step S3 of Figure 5. The feature space 50 includes multiple feature vectors va, each transformed using a product model from images of multiple products corresponding to product information stored in the memory device 23. A portion of the feature vectors va corresponds to product A (indicated by "○"), and another portion of the feature vectors va corresponds to product B (indicated by "□"). The feature space 50 also includes feature vectors v, which are transformed from the input image using the product model.

[0056] The processor 21 determines at least one feature vector va from among multiple feature vectors va that is similar to feature vector v. Determining at least one feature vector va similar to feature vector v involves performing k-nearest neighbor search in the feature space 50. The processor 21 sets up k-nearest neighbors 51 around feature vector v, each containing a predetermined k number of feature vectors va. In the example in Figure 7, k=3. The k-nearest neighbors 51 include two feature vectors va ("○") corresponding to product A and one feature vector va ("□") corresponding to product B. The processor 21 estimates the product class corresponding to feature vector v by aggregating the labels of the feature vectors va included in the k-nearest neighbors 51. In the example in Figure 7, it is estimated that feature vector v corresponds to product A with a probability of 66% and to product B with a probability of 33%. By using k-nearest neighbor search, high-dimensional feature vectors output from a neural network can be processed quickly and accurately.

[0057] Figure 8 shows the candidate product information displayed on the terminal device 3 in step S5 of Figure 5. The terminal device 3 displays information 60 on the display device 36. The information 60 includes an input image 61, search conditions 62, and search results 63. The information 60 may further include a purchase price 64.

[0058] The input image 61 includes a magnified image 61a and a thumbnail image 61b of the photographed product 6. If multiple input images showing the same product 6 are provided, the appraiser can change the magnified image 61a by selecting the desired image from the thumbnail images 61b.

[0059] The search criteria 62 indicates the label of the product estimated to correspond to product 6 in the input image 61. The input field 62a contains each item of the label. The input field 62a is automatically filled in by the processor 31 based on the estimated label. The input field 62a may be automatically filled in when the user presses the completion button 62b, or it may be automatically filled in as a trigger for other appropriate events. If the contents of the input field 62a are incorrect, or if the input field 62a is blank, the assessor enters the correct information into the input field 62a and presses the completion button 62b. When the completion button 62b is pressed, the processor 31 sends the contents of the input field 62a to the server device 2.

[0060] Search results 63 include candidate product information 63-1 to 63-3 that are estimated to correspond to product 6 in the input image. Candidate product information 63-1 to 63-3 are displayed in order of likelihood of corresponding to product 6 in the input image. Candidate product information 63-1 includes the product image 63-1a, description 63-1b, and transaction history 63-1c, and candidate product information 63-2 to 63-3 also include similar information to candidate product information 63-1. Note that transaction history 63-1c may be displayed as part of candidate product information 63-1, or it may be displayed on a screen accessed by selecting one of the candidate product information 63-1 to 63-3 in the search results.

[0061] The purchase price 64 indicates the amount obtained as a result of assessing product 6 in the input image. Input field 64a contains the assessed amount. Input field 64a may be entered by the assessor. Alternatively, input field 64a may be automatically entered by the processor 31 based on the transaction history 63-1c of the candidate product information 63-1. If the transaction history 63-1c is managed by the condition of the product (good, dirty, scratched, junk, etc.), the processor 31 may enter the amount determined according to the condition of product 6 into input field 64a. Input field 64a may be automatically entered when the user presses the completion button 64b, or it may be automatically entered triggered by other appropriate events. If the content of input field 64a is undesirable, or if input field 64a is blank, the assessor enters the purchase price into input field 64a and presses the completion button 64b. When the completion button 64b is pressed, the processor 31 sends the content of input field 64a to the server device 2. Note that the purchase price 64 may not be included on the same screen as the input image 61, search criteria 62, and search results 63, but may be displayed on a different screen accessed via a transition.

[0062] By displaying the information 60 in Figure 7, the terminal device 3 eliminates or reduces the need for the appraiser to manually input search criteria, thereby supporting the appraiser's assessment of product 6.

[0063] The display device 36 displays the input image 61 and the image of the product included in the candidate product information (e.g., 63-1a) in a comparable manner. For example, as shown in Figure 8, the display device 36 may display the input image and the image of the product included in the candidate product information simultaneously. Alternatively, the display device 36 may alternately display the input image and the image of the product included in the candidate product information via one or more screen transitions. This can support the visual confirmation by the appraiser.

[0064] Figure 9 shows the appraisal information displayed on terminal device 1 in step S7 of Figure 5. Terminal device 1 displays information 70 on display device 16. Information 70 includes the image 63-1a and description 63-1b of the product in Figure 8, and the purchase price determined (or approved) by the appraiser. The sales representative presents the purchase price displayed on display device 16 to the owner of product 6 and proceeds with the transaction for product 6.

[0065] [Product Model Training] The product model 40 is trained to output different feature vectors when different product images are input, using a data set containing various images of various products and associated product information. To train the product model 40, a subset of images randomly sampled from the data set is used as training data. For example, the training data contains an average of 15 images per product (sample). This ensures that even when training the same product, the combination and number of images input to the product model 40 will be different each time. This is equivalent to data augmentation of the data set.

[0066] The convolutional neural network 41 and set transformer 42 of product model 40 are trained simultaneously.

[0067] The product model 40 is trained using a predetermined loss function, which represents the distance in the feature space. The parameters of the product model 40 are set (i.e., trained) so that feature vectors corresponding to similar products are close to each other in the feature space, and feature vectors corresponding to different products are far apart in the feature space, based on the equivalence relations of the product information (same / different types, same / different names, same / different model numbers, etc.). In this case, the product information is used as metadata for the feature vectors. The product information may be managed as integer data, or it may have any other arbitrary type that allows for the definition of equivalence relations.

[0068] In this embodiment, the product model 40 is trained using the sum of two loss functions described below.

[0069] Figure 10 illustrates the first loss function used when training the product model 40 in Figure 6. Figure 10 shows the calculation of the loss function based on kNN loss. For example, a general loss function based on cross-entropy reflects the distance between the point of interest and the representative point (prototype) of each class. In this case, the distance between the point of interest and points other than the representative point is ignored. On the other hand, in this embodiment, labels are determined by k-nearest neighbor search rather than class probability during inference, so a corresponding loss is introduced during training. The loss function based on kNN loss calculates the loss based on the distance between points (i.e., feature vectors) in the feature space. The loss function based on kNN loss reflects the distance between the point of interest 81 and all other points 82. The coordinates of all points in the feature space are determined by the current parameters set for the product model 40. As training progresses, the parameters of the product model 40 are updated so that the coordinates of all points move slightly. However, to reduce the computational load, the parameters of the product model 40 may be updated at each training step so that only the point of interest moves. When updating the parameters of product model 40, the parameters may be stored and processed separately from product model 40 until the update is complete. The loss function based on kNN loss is well-suited for distance-based inference tasks such as k-nearest neighbor search.

[0070] Figure 11 illustrates the second loss function used when training the product model 40 in Figure 6. Figure 11 shows the calculation of the loss function based on the control loss (Npairs loss). If the target task is classification to subcategories (i.e., with fine granularity), learning only the major categories (i.e., with coarse granularity) labels may result in insufficient resolution for the product model 40 to labels. Learning supplements the ability to extract fine features by assigning the task of discriminating the object of interest itself, in addition to classification to major categories. Sets of input images 43-A to 43-C are used to train the product model 40. Sets 43-A and 43-B represent the same product (although at least some elements of the set are different), and set 43-C represents a different product from the products in sets 43-A and 43-B. Feature vectors v1 to v3 corresponding to sets 43-A to 43-C are calculated using the product model 40. Feature vectors v1 and v2 correspond to the same product, so it is desirable that they be as close as possible or even identical. Similarly, feature vector v3 corresponds to a different product than those represented by feature vectors v1 and v2, so it is desirable that it be as far apart as possible. These states are evaluated by the distance between two points in the feature space. The parameters of the product model 40 are adjusted based on the loss function so that feature vectors v1 and v2 are closer to each other, and feature vector v3 is further away from feature vectors v1 and v2. This allows for highly accurate determination of whether a set of input images represents the same product or not. Here, the set of input images may be associated with labels of the same class or with labels of different classes. Therefore, it is possible to distinguish differences finer than those represented by labels.

[0071] Product model 40 is trained in units of minibatches. For each sample in a minibatch, samples other than itself correspond to different products. In other words, since samples corresponding to different products are naturally obtained when minibatch training is adopted, all that remains is to prepare other samples corresponding to the same product to obtain triplets for comparison. The input data in one training step basically consists of two sets of input images with different data augmentation for each sample, and corresponding product information (labels). By collecting multiple sets of input data to form minibatches, it is possible to have both knn loss and control loss coexist.

[0072] [Generating the feature space] To generate the feature space, a data set containing product information with a different granularity than the data set used to train the product model 40 may be used. Here, the granularity of the product information indicates the degree of detail of the product features. For example, coarse-grained product information may include product types such as bags, clothing, shoes, etc., while fine-grained product information may include product types such as handbags, shoulder bags, backpacks, etc.

[0073] In this case, the processor 21 trains the product model 40 by inputting images associated with product information having a first granularity, i.e., images included in a data set containing coarse-grained product information, into the product model 40. The processor 21 also generates a feature space by inputting images associated with product information having a second granularity, which is finer than the first granularity, i.e., images included in a data set containing fine-grained product information, into the product model 40.

[0074] When a new class label is added to the data set, the feature space is updated in the following way: First, samples (images and product information) with the new class label are added to the data set containing fine-grained product information. Next, the added samples are converted into feature vectors and projected onto the feature space. At this time, the product model 40 is not updated (i.e., no further training of the product model 40 is performed). Since the updated feature space includes the new class label, it is possible to return the new class label during inference.

[0075] [Effects of the Embodiment] According to the embodiment, the need for sales representatives and appraisers to manually input product information is eliminated or reduced, and the appraiser can be assisted in appraising product 6. According to the embodiment, product 6 can be appraised with less time and effort than before.

[0076] Traditionally, sales representatives or appraisers have had to manually enter product information into a search box, which has been time-consuming and laborious. Since appraisers appraise dozens of products a day, the time spent on data entry adds up to a considerable loss of time. On the other hand, according to this embodiment, the time required to identify product information for product 6 can be significantly reduced.

[0077] According to one embodiment, by using machine learning image recognition, product-identifiable information can be automatically extracted from product images, and candidate product information can be presented to the appraiser in a list format. The appraiser can complete product identification simply by selecting the correct product from the presented candidate product information, for example, by clicking using a pointing device.

[0078] Generally, the classes to be classified have a high degree of visual similarity to each other, and there is an extreme bias in the number of samples for each class. For this reason, general classification algorithms based on cross-entropy tend to have a high rate of misclassification, especially for minor classes. On the other hand, according to this embodiment, similarity search is performed on the feature space, and the labels assigned to neighboring samples are aggregated to determine the final output. This makes it possible to reduce the rate of misclassification for classes with high visual similarity and for minor classes.

[0079] The number of product classes subject to assessment may increase or decrease due to the addition of new products or the discontinuation of existing products. In this embodiment, to reduce the effort required for retraining due to changes in class categories, feature extraction and label learning are separated, and furthermore, label learning is implemented without parameters. Handling class increases or decreases only requires re-running the label learning process, which can be achieved with very low computational cost.

[0080] Generally, when adding a new product category to the recognition target, it is necessary to train based on the new data set. Assigning annotations to product images is a task of comparable difficulty to appraisal work, requiring advanced expertise and therefore being very costly. On the other hand, according to the embodiment, this problem is addressed by separating feature extraction and label learning. In this two-stage process, consistency of labels in the feature extraction unit and the label learning unit is not required. The feature extraction unit requires a large number of samples for training, but the label granularity can be coarser than in operation. Label learning requires labels of the same granularity as in operation, but since the model is lightweight, it requires less computation. In other words, the labels required to train the proposed model are a sufficient number of broad category labels and a small number of subcategory labels, and the cost of collecting these is very inexpensive compared to assigning subcategory labels to all samples.

[0081] When assessing a single product, the system receives multiple images of that product taken from various angles. According to this embodiment, multiple input images can be processed simultaneously in a single inference. By enabling simultaneous input, the model dynamically determines the importance of the input images while making predictions. The processing time is approximately constant with respect to the number of input images, achieving both the utilization of increased information from multiple input images and stable throughput.

[0082] Generally, a system receives multiple images of a product taken from various angles, but the order and number of images are not fixed, and missing and / or duplicate images occur. For example, two images may be input in the order of overhead → close-up, or three images may be input in the order of close-up → close-up → overhead. Constraints on the order and number of input images limit the operations of sales representatives and appraisers. On the other hand, according to the embodiment, this problem is addressed by implementing a model in which the output feature vector does not depend on the order and / or number of input images. According to the embodiment, since the constraints on the order and number of input images are eliminated, machine learning can be introduced without changing the operations at the appraisal site at all.

[0083] According to this embodiment, the terminal device 3 displays the search results the moment the appraiser proceeds to the appraisal screen. Furthermore, according to this embodiment, the terminal device 3 displays the candidate product information selected by the appraiser and the captured image in a comparable manner. Additionally, according to this embodiment, when candidate product information is selected, the terminal device 3 fills in the information in the search box. This significantly reduces the workload for the appraiser.

[0084] [Other embodiments] Server device 2 and terminal device 3 may be integrated with each other.

[0085] The candidate product information presented to the appraiser may include product model numbers and other details.

[0086] The data set used to train the product model 40 and the data set used to generate the feature space may share at least some images. Alternatively, a common data set may be used to train the product model 40 and generate the feature space.

[0087] Instead of k-nearest neighbor search, any other multi-class classification algorithm that converts features into labels may be used to search for similar feature vectors in the feature space. [Industrial applicability]

[0088] An appraisal support device and appraisal support system according to one aspect of this disclosure are applicable to appraising goods for purposes such as buying and selling used goods. [Explanation of Symbols]

[0089] 1 Terminal device, 2 Server device, 3 Terminal device, 5 Communication line, 6 Product, 11 Processor, 14 Communication device, 16 Display device, 17 Camera, 21 Processor, 23 Storage device, 24 Communication device, 31 Processor, 34 Communication device, 35 Input device, 36 Display device, 40 Product model, 41 Convolutional neural network (CNN), 42 Set transformer, 43-1~43-4 Input image.

Claims

1. A communication device connected to the first and second terminal devices via a communication line, A storage device that stores product information for multiple products, Equipped with a processor, The aforementioned processor, Train the product model to output different feature vectors when different product images are input. The first terminal device receives at least one input image representing one product, Using the aforementioned product model, the input image is converted into a first feature vector, Using the aforementioned product model, images of multiple products corresponding to the product information stored in the storage device are converted into multiple second feature vectors. From the plurality of second feature vectors, at least one second feature vector similar to the first feature vector is determined. Product information corresponding to a second feature vector similar to the first feature vector is read from the storage device as candidate product information corresponding to the product in the input image. The candidate product information is transmitted to the second terminal device. Training the product model includes inputting images associated with product information having a first level of granularity into the product model. Converting the image of the product into the second feature vector includes inputting the image associated with product information having a second granularity finer than the first granularity into the product model. Assessment support device.

2. The aforementioned product model includes a convolutional neural network and a set transformer. The assessment support device according to claim 1.

3. The processor determines at least one second feature vector similar to the first feature vector by performing a k-nearest neighbor search in a vector space containing the first and second feature vectors. The assessment support device according to claim 1.

4. The processor receives appraisal information of the product in the input image from the second terminal device and transmits the appraisal information to the first terminal device. The assessment support device according to claim 1.

5. The first terminal device and, The second terminal device and, Includes an assessment support device according to one of claims 1 to 4, Assessment support system.

6. An appraisal support system including a first terminal device, a second terminal device, and an appraisal support device connected to the first and second terminal devices via a communication line, The first terminal device comprises a camera, a first display device, and a first processor. The second terminal device comprises a second display device, an input device, and a second processor. The appraisal support device comprises a storage device for storing product information relating to multiple products and a third processor. The first processor is, The camera generates at least one input image representing one product, The input image is transmitted to the assessment support device. The third processor described above is Train the product model to output different feature vectors when different product images are input. The input image is received from the first terminal device. Using the aforementioned product model, the input image is converted into a first feature vector, Using the aforementioned product model, images of multiple products corresponding to the product information stored in the storage device are converted into multiple second feature vectors. From the plurality of second feature vectors, at least one second feature vector similar to the first feature vector is determined. Product information corresponding to a second feature vector similar to the first feature vector is read from the storage device as candidate product information corresponding to the product in the input image. The candidate product information is transmitted to the second terminal device. Training the product model includes inputting images associated with product information having a first level of granularity into the product model. Converting the image of the product into the second feature vector includes inputting the image associated with product information having a second granularity finer than the first granularity into the product model. The second processor is, The appraisal support device receives the candidate product information, The candidate product information is displayed on the second display device. The product assessment information of the input image is obtained via the input device. The assessment information is transmitted to the assessment support device. The third processor described above is The assessment information is received from the second terminal device. The assessment information is transmitted to the first terminal device. The first processor is, The assessment support device receives the assessment information, The assessment information is displayed on the first display device. The aforementioned candidate product information includes the product's image, name, and past transaction history. Assessment support system.

7. The second display device displays the input image and the image of the product included in the candidate product information in a comparable manner. The assessment support system according to claim 6.

8. The third processor recognizes a plurality of input images representing a single product, and reads from the storage device at least one set of product information corresponding to one of the products in the input images as candidate product information. The assessment support system according to claim 6 or 7.

Images