Information processing method, program, and information processing system

Abstract

To enable a scratch or the like of an article to be detected without preparing an image of the article in advance.SOLUTION: An information processing method is an information processing method to be executed by an information processing device, and includes: an image acquisition step of acquiring a photographic image; a division step of performing division in each part of the article from the photographic image; a detection step of detecting a scratch of the article from the photographic image on the basis of a learned scratch model; and a superimposition step of superimposing the part divided at the division step on the scratch detected at the detection step.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Cited Document 1 discloses a damage determination device that detects scratches on an article by comparing the image with an image of a genuine article. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6954741 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the above-mentioned prior art, it is necessary to obtain an image of the genuine article in advance, and it is difficult to detect scratches and the like unless an image of the genuine article is prepared in advance.

[0005] In consideration of the above, an object of the present invention is to provide an information processing method, a program, and an information processing system that are capable of detecting scratches and the like on an article without preparing an image of the article in advance. [Means for solving the problem]

[0006] According to one embodiment, the information processing method is executed by an information processing device and includes an image acquisition step of acquiring a captured image, a division step of dividing the captured image into parts of the item, a detection step of detecting scratches on the item from the captured image based on a learned scratch model, and an overlay step of overlaying the parts divided in the division step and the scratches detected in the detection step.

[0007] According to a program of one embodiment, an information processing device is caused to execute an information processing method including an image acquisition step of acquiring a captured image, a division step of dividing the captured image into parts of the item, a detection step of detecting scratches on the item from the captured image based on a learned scratch model, and an overlay step of overlaying the parts divided in the division step and the scratches detected in the detection step.

[0008] According to one embodiment, the information processing system is executed by an information processing device and includes an image acquisition step of acquiring a captured image, a division step of dividing the captured image into parts of the item, a detection step of detecting scratches on the item from the captured image based on a learned scratch model, and an overlay step of overlaying the parts divided in the division step with the scratches detected in the detection step. Effect of the Invention

[0009] According to one embodiment, scratches and the like on an article can be detected without preparing an image of the article in advance. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal according to an embodiment. [Figure 6] 5 is a flowchart illustrating an example of processing executed by the information processing system according to the embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a screen when photographing an article according to the embodiment. [Figure 8]FIG. 13 is a diagram showing an example of an image of an article divided into parts according to the embodiment. [Figure 9] 5 is a flowchart illustrating an example of processing executed by the information processing system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0012] (System Overview) First, an overview of an information processing system 10 according to the present embodiment will be described. The information processing system 10 according to the present embodiment is a system that can divide an object such as an automobile into parts from a photographed image and detect scratches, etc. In the present embodiment, the information processing system 10 is a system that can divide an object into parts from a photographed image of the object by a user, detect scratches, etc. from the photographed image, and further evaluate the value of each part of the object and the value of the object as a whole.

[0013] (System Configuration) Fig. 1 is a diagram showing an example of a configuration of an information processing system 10 according to the present embodiment. As shown in Fig. 1, the information processing system 10 according to the present embodiment includes a user terminal 2 and a server 1, which are communicably connected to each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination of these.

[0014] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these. In this embodiment, an information processing device equipped with a photographing function is used as an example.

[0015] The server 1 is an example of an information processing device that acquires an image of an article photographed by a user terminal 2, disassembles the article into individual parts in the photographed image, detects scratches or the like on the article, and outputs the results. The server 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server 1 will be described later.

[0016] (Hardware configuration - Server) 2 is a block diagram showing a hardware configuration of the server 1. The server 1 includes a processor 201, a memory 202, a storage 203, and a communication I / F 204, which are communicatively connected to each other via a bus B.

[0017] The processor 201 controls each component of the server 1 and realizes the functions of the server 1 by expanding various programs stored in the storage 203 into the memory 202 and executing them. The programs executed by the processor 201 include, but are not limited to, an OS (Operating System) and various programs described below. The execution of these programs by the processor 201 realizes a part of the state visualization method according to this embodiment. The processor 201 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination of these.

[0018] The memory 202 is, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), or a combination thereof. The ROM is, for example, a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), or a combination thereof. The RAM is, for example, a Dynamic RAM (DRAM), a Static RAM (SRAM), a Magnetoresistive RAM (MRAM), or a combination thereof.

[0019] The storage 203 stores an OS, various programs described below, and various data. The storage 203 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0020] The communication I / F 204 is an interface for connecting the server 1 to external devices including the user terminal 2 and the image capturing device 16 via the network N and for controlling communication. The communication I / F 204 is, for example, an adapter conforming to Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (for example, Fibre Channel), but is not limited thereto.

[0021] (Hardware configuration - User terminal) 3 is a block diagram showing a hardware configuration of the user terminal 2. The user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, an input device 206, and an output device 207, which are communicatively connected to each other via a bus B. The communication I / F 204 is an interface for connecting the user terminal 2 to an external device including the server 1 via a network N and controlling communication. The communication I / F 204 is, for example, an adapter conforming to Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (for example, Fibre Channel), but is not limited thereto.

[0022] The input / output I / F 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, an operation button, or a combination of these. The output device 207 as a user interface is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination of these. In this embodiment, as an example, the output device 207 and the input device 206 are an integrally configured touch panel display.

[0023] In this embodiment, the program may be written in the memory 202 or the storage 203 during the manufacturing stage of the server 1, or may be provided to the server 1 via the network N. Also, the program may be provided to the server 1 via a non-transitory computer-readable recording medium such as a disk medium (not shown).

[0024] (Functional configuration - Server) Next, the functional configuration of the server 1 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the server 1. When executing various programs, the server 1 realizes various functions using the above hardware resources. The server 1 has a communication unit 11, a storage unit 12, and a control unit 13 as the functional configuration realized by the server 1. Each functional configuration is realized by the processor 101 reading and executing a program 121 stored in the memory 102 or storage 103.

[0025] The control unit 13 includes an image acquisition unit 131 that acquires an image captured by the user terminal 2, a division unit 132 that divides the image of the item acquired by the image acquisition unit 131 into parts of the item, and a detection unit 133 that detects scratches, etc. on the item from the image acquired by the image acquisition unit 131. The control unit 13 further includes an overlapping unit 134 that overlaps the scratches, etc. detected by the detection unit 133 for each part divided by the division unit 132, and a determination unit 135 that determines the value of the entire item and the value of each part from the images overlapped by the overlapping unit 134. These will be described in detail later.

[0026] (Functional configuration - User terminal) Next, the functional configuration of the user terminal 2 will be described. Fig. 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 realizes various functions using the above hardware resources. The user terminal 2 has a communication unit 21, a storage unit 22 in which a program 221 is stored, and a control unit 23 as functional configurations realized by the user terminal 2. Each functional configuration is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or the storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server 1, and a display unit 232 that displays the acquired information on the output device 207.

[0027] Next, the flow of processing of the information processing system according to this embodiment will be described with reference to Figs. 6 to 8. Fig. 6 is a diagram relating to processing executed by the information processing system 10 according to this embodiment. Fig. 7 is a schematic diagram showing an example of a screen displayed on the user terminal 2. Fig. 8 is a schematic diagram showing a state in which an article is divided into parts.

[0028] First, in step S101, the user terminal 2 transmits a photographed image of an item to the server 1, and the image acquisition unit 131 acquires the photographed image. Next, in step S102, the server 1 determines whether or not the entire item has been photographed. In this embodiment, when photographing an item, it is necessary to photograph the entire item. That is, the user terminal 2 photographs the item from all directions to acquire a photographed image of the entire item. In this embodiment, an automobile is used as an example of an item, and photographed images are acquired from eight directions, including the front left diagonal, front right diagonal, rear left diagonal, and rear right diagonal, in addition to the front, back, left, right, and rear diagonal directions of the automobile. Note that, even for other items, photographed images may be acquired from other directions, such as the top and bottom, in addition to the front, back, left, and right diagonal directions, as necessary.

[0029] Here, photographing an object will be described with reference to FIG. 7. FIG. 7 is an example of a screen displayed on the display unit of the user terminal 2 when photographing an object. Screen IM1 displays eight screens T1 to T8 required for photographing. Screens T1 to T8 display the photographed object, and correspond to the photographing directions t1 to t8 displayed on screen IM2. Before photographing an object, nothing is displayed on screens T1 to T8, and the photographing directions t1 to t8 on screen IM2 are also white. In this embodiment, the object is an automobile, so the photographing directions are t1 to t8. However, if the object is something other than an automobile, the object to be photographed on screen IM2 will be different, and the photographing directions t1 to t8 will also be different depending on the object.

[0030] In addition, the shooting directions t1 to t8 also function as a guide for shooting, and not only indicate the direction from which shooting should be performed, but also turn black when shooting has already been performed, so that it is clear at a glance whether shooting has been performed or not. Specifically, the screen T1 corresponds to the shooting direction t1, and since shooting has already been performed, the shot image is displayed on the screen T1, and the shooting direction t1 is blacked out. In contrast, the screen T2 is before shooting, and the corresponding shooting direction t2 is white. When the shot image is displayed on all the screens T1 to T8 and the shooting directions t1 to t8 are blacked out, shooting is completed, and the completion button IM3 is pressed to complete shooting.

[0031] In this embodiment, a schematic diagram of the target object and shooting directions t1 to t8 are displayed on the screen IM2 as a guide when shooting an object, but the present invention is not limited to this, and may be, for example, displayed as text or displayed on the shooting screen. Specifically, when shooting an object is started on the user terminal 2, the shooting direction, etc. may be displayed on the screen, and when one shot is completed, instructions for the next shot may be displayed. As a guide during shooting, shooting guide may be not only displayed on the screen, but also audio. Furthermore, in this embodiment, when shooting has already been performed, the image is filled in black, but the present invention is not limited to this, and may be filled in with a color other than black, and may be displayed in any way that allows the user to determine whether shooting has already been performed or not.

[0032] Next, in step S102, when all the photographing is completed, that is, when the photographed images are displayed on all the screens T1 to T8, the photographing directions t1 to t8 are all filled in black, and the button IM3 is pressed, the process proceeds to step S103. Next, in step S103, the division unit 132 divides the article into parts from the photographed image acquired in step S101. Here, FIG. 8 shows an example of an image of an article divided into parts. FIG. 8 shows an example of a diagram in which a front image of an automobile 3, which is an article of this embodiment, is divided. As shown in FIG. 8, the automobile 3 is divided into parts M1 to M8. In this embodiment, the automobile 3 is divided into parts M1 to M8, but for example, the parts to be divided may be selected according to the article. In the automobile 3 of this embodiment, the front is divided into a windshield, side mirrors, headlights, front grille, and tires. Although not shown, the automobile 3 is divided into side mirrors, doors, windows, door handles, and tires on the side, and into rear windshield, backlights, rear bumper, and back door on the back.

[0033] The division of parts is not limited to this and may be changed as appropriate. If the article is other than an automobile, it is divided into parts according to the article. In this embodiment, the article is divided from a learning model of the parts in the article. The automobile 3 in this embodiment is divided into parts M1 to M8 based on a learning model of the automobile article from a photographed image. In addition to the learning model, feature points may be extracted from the photographed image and divided into each part from the feature points.

[0034] Next, in step S104, the detection unit 133 detects scratches, etc. of the object from the captured image acquired in step S101. In this embodiment, scratches, etc. are detected based on a learning model of scratches, etc. In this embodiment, scratches, etc. of an automobile will be described as an example of an object. Scratches, etc. on an automobile are, for example, scratches, scuffs, and dents, and the detection unit 133 detects scratches, etc. from the learning model of these scratches. In the case of other objects, scratches, etc. on the object are scratches including defects, damage, and dents. Since scratches, etc. on the object are detected from the learning model of scratches, there is no need to obtain an image of a genuine object in advance. Therefore, scratches can be detected regardless of the type of vehicle. In other words, when detecting scratches, etc. by comparing with an image of a genuine object, an image of the genuine object is required in advance, and it is difficult to detect scratches, etc. without an image of a genuine object, but by detecting scratches, etc. from the learning model of scratches, it is possible to detect scratches, etc. on any object.

[0035] Next, in step S105, the superimposing unit 134 superimposes the parts divided in step S103 and the scratches, etc. detected in step S104, and outputs the superimposed image in step S106. Specifically, in this embodiment, as shown in part M3 in FIG. 8, the detected scratches and parts are superimposed, and the parts on which scratches, etc. are detected are displayed so as to be emphasized. Note that, without being limited to this, the display may be changed to make it easier to understand, such as by hatching the parts on which scratches, etc. are detected. In addition, in this embodiment, only the front image is shown, but the superimposed images are also displayed in the same manner for the side and back images. As described above, scratches, etc. of the object can be detected without preparing a normal image of the object in advance. Note that the order of steps S103 and S104 may be reversed, and the processes may be carried out in parallel. In addition, in this embodiment, the results of the superimposition in step S105 are displayed in step S106, but in addition to being displayed, these may be stored in a recording medium such as a database and stored as the state of the object at the time of shooting.

[0036] FIG. 9 is a diagram showing a process including a step of determining the overall value of the item and the value of each part. The same process as in FIG. 6 will not be described, and only the different parts will be described. After the overlapping in step S105, in step S107, the determination unit 135 determines the overall value of the item and the value of each part. By determining the overall value of the item and the value of each part, it is possible to know whether the item should be sold as a whole or disassembled into parts and sold in the secondhand market. Specifically, the determination unit 135 can know which part has scratches, etc. in the images overlapped in step S105, and can therefore know the cost required for repair due to the scratches, etc. That is, in the automobile 3 of this embodiment, when there is a scratch on the part M3 as shown in FIG. 8, it is possible to determine, for example, that there is a scratch on the side mirror, and therefore it is possible to determine how much cost is required for the scratch on the side mirror.

[0037] Even if scratches or the like have been repaired before the image was taken, the determination unit 135 determines the value from an external repair history database. Regarding the value determination of the whole item, for example, by linking with an external database, the model can be specified and the value in the secondhand market can be known. Similarly, by linking with an external database, the secondhand market value of each part can be determined. In this way, the determination unit 135 can determine the value of the whole item (in this embodiment, an automobile) and the value of each part from the state of scratches or the like, the parts, the model, the past repair history, etc., by linking with the photographed image acquired in step S101 and the external database. Then, in step S106, not only the superimposed images but also the result of the value determined in step S107 are output together. In addition to outputting as described above, when the state of the item is saved, the state of the item can be saved in chronological order, and the change in the value of the item in the market can also be known.

[0038] As described above, according to this embodiment, scratches and the like on an item can be detected without preparing a normal image of the item in advance. In addition, since scratches and the like can be detected for each part, the value of the item as a whole and the value of each part can also be evaluated. Furthermore, by linking with an external database, the value can be evaluated more clearly. In other words, regardless of the type of item, the value of the item can be automatically evaluated by simply photographing the item itself.

[0039] In addition, in this embodiment, an automobile is described as an example of an item, but other items can be similarly subjected to detection of scratches and the like for each part and value evaluation of the item. When the item is an automobile as in this embodiment, the automobile is bought and sold as a whole body or as individual parts in the secondhand market. Therefore, depending on the type of car, it is difficult to know in advance whether it is appropriate to buy and sell the entire body or whether it is appropriate to dismantle it into individual parts. However, according to this embodiment, the value of the car body and the value of each part can be known in advance by photographing the car on the spot regardless of the type of car. In addition, even in the case of an auction between consumers, both the seller and the buyer can make a transaction with peace of mind by photographing the car to detect scratches and the like and determine the value.

[0040] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope that can achieve the object of the present invention are included in the present invention.

[0041] Also, for example, the above-mentioned series of processes can be executed by hardware or software. In other words, the functional configuration is merely an example and is not particularly limited. That is, it is sufficient that the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and there is no particular limit to what kind of functional block is used to realize this function. In addition, the location of the functional block is not particularly limited and may be arbitrary. For example, the functional block of the server may be transferred to a user terminal or the like. Conversely, the functional block of the user terminal may be transferred to a server or the like. In addition, one functional block may be configured by hardware alone, software alone, or a combination thereof.

[0042] For example, when a series of processes is executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0043] Furthermore, for example, a recording medium containing such a program may not only be constituted by a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but may also be constituted by a recording medium that is provided to the user in a state in which it is pre-installed in the device main body.

[0044] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically according to the order, but also processes that are not necessarily performed chronologically but are executed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is composed of a plurality of devices, a plurality of means, etc. [Explanation of symbols]

[0045] 10 Information Processing Systems 2 User terminal (information processing device) 1 Server (information processing device) 131 Image Acquisition Unit 132 divisions 133 Detector 134 overlapping section

Claims

1. An information processing method executed by an information processing device, An image acquisition step of acquiring a photographed image; A dividing step of dividing the photographed image into parts of the article; A detection step of detecting a scratch on the article from the captured image based on a learned scratch model; a superposition step of superposing the parts divided in the dividing step and the flaw detected in the detecting step; An information processing method comprising:

2. A guide step of transmitting information for guiding a photographing direction in which a photographed image of the entire article can be obtained when the photographed image is obtained. Also includes The information processing method according to claim 1 .

3. A determination step for determining the overall value of the article and the value of each part of the article from the images of each part superimposed in the superimposing step. Also includes The information processing method according to claim 1 .

4. The damage includes at least a defect, a damage, and a dent in the article. The information processing method according to any one of claims 1 to 3.

5. In the information processing device, An image acquisition step of acquiring a photographed image; A dividing step of dividing the photographed image into parts of the article; A detection step of detecting a scratch on the article from the captured image based on a learned scratch model; and an overlapping step of overlapping the parts divided in the dividing step and the flaw detected in the detecting step.

6. An information processing system executed by an information processing device, An image acquisition step of acquiring a photographed image; A dividing step of dividing the photographed image into parts of the article; A detection step of detecting a scratch on the article from the captured image based on a learned scratch model; a superposition step of superposing the parts divided in the dividing step and the flaw detected in the detecting step; An information processing system that performs the above.

Images