Information processing methods, programs, and information processing systems

Abstract

To enable a scratch or the like of an article to be detected without preparing an image of the article in advance.SOLUTION: An information processing method is an information processing method to be executed by an information processing device, and includes: an image acquisition step of acquiring a photographic image; a division step of performing division in each part of the article from the photographic image; a detection step of detecting a scratch of the article from the photographic image on the basis of a learned scratch model; and a superimposition step of superimposing the part divided at the division step on the scratch detected at the detection step.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing system.

Background Art

[0002] Citation Document 1 discloses a damage determination device that detects damage such as scratches on an article by comparing it with an image of a regular article.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of the above prior art, it is necessary to acquire an image of a regular article in advance, and it is difficult to detect damage such as scratches if an image of a regular article is not prepared in advance.

[0005] In consideration of the above facts, an object of the present invention is to obtain an information processing method, a program, and an information processing system that can detect damage such as scratches on an article without preparing an image of the article in advance.

Means for Solving the Problems

[0006] According to an information processing method according to an embodiment, it is an information processing method executed by an information processing device, including an image acquisition step of acquiring a photographed image, a division step of dividing the photographed image for each part of an article, a detection step of detecting damage of the article from the photographed image based on a learned damage model, and an overlapping step of overlapping the parts divided in the division step and the damage detected in the detection step.

[0007] According to a program according to one embodiment, the information processing device is made to execute an information processing method that includes an image acquisition step of acquiring a captured image, a division step of dividing the captured image into parts of an article, a detection step of detecting scratches on the article from the captured image based on a learned scratch model, and a superposition step of superimposing the parts divided in the division step and the scratches detected in the detection step.

[0008] According to an information processing system according to one embodiment, the information processing system performed by the information processing device includes an image acquisition step of acquiring a captured image, a division step of dividing the captured image into parts of an article, a detection step of detecting scratches on the article from the captured image based on a learned scratch model, and a superposition step of superimposing the parts divided in the division step and the scratches detected in the detection step. [Effects of the Invention]

[0009] According to one embodiment, scratches or other defects on an item can be detected without having to prepare an image of the item in advance. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of an information processing system according to the embodiment. [Figure 2] This figure shows an example of the server hardware configuration according to the embodiment. [Figure 3] This figure shows an example of the hardware configuration of a user terminal according to the embodiment. [Figure 4] This figure shows an example of the functional configuration of the server according to the embodiment. [Figure 5] This figure shows an example of the functional configuration of a user terminal according to the embodiment. [Figure 6] This flowchart shows an example of a process performed by the information processing system according to the embodiment. [Figure 7] This figure shows an example of a screen when photographing an article according to the embodiment. [Figure 8]This figure shows an example of an image of an article divided into parts according to the embodiment. [Figure 9] This flowchart shows an example of a process performed by the information processing system according to the embodiment. [Modes for carrying out the invention]

[0011] An embodiment of the information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each figure, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0012] (System Overview) First, an overview of the information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that can divide an image of an item, such as an automobile, into individual parts and detect scratches and other defects. In this embodiment, the information processing system 10 is a system that can divide an image of an item taken by a user into individual parts, detect scratches and other defects from the image, and further evaluate the value of each part of the item and the value of the item as a whole.

[0013] (System Configuration) Figure 1 shows an example of the configuration of the information processing system 10 according to this embodiment. As shown in Figure 1, the information processing system 10 according to this embodiment includes a user terminal 2 and a server 1 that are connected to each other so as to be able to communicate with each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public telephone network, a mobile data communication network, or a combination thereof.

[0014] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various types of information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination thereof. In the present embodiment, an information processing device equipped with a photographing function is taken as an example.

[0015] The server 1 is an example of an information processing device that acquires a photographed image of an article photographed by the user terminal 2, decomposes the article into parts in the photographed image, detects damages and the like of the article, and outputs the results. The server 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination thereof. The specific configuration and operation of the server 1 will be described later.

[0016] (Hardware Configuration - Server) FIG. 2 is a block diagram showing the hardware configuration of the server 1. The server 1 includes a processor 201, a memory 202, a storage 203, and a communication I / F 204, which are communicably connected to each other via a bus B.

[0017] The processor 201 controls each component of the server 1 and realizes the functions of the server 1 by expanding and executing various programs stored in the storage 203 in the memory 202. The programs executed by the processor 201 include, but are not limited to, an OS (Operating System) and various programs described later. By the processor 201 executing these programs, a part of the state visualization method according to the present embodiment is realized. The processor 201 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

[0018] The memory 202 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or a combination thereof. The ROM is, for example, a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), or a combination thereof. The RAM is, for example, a DRAM (Dynamic RAM), a SRAM (Static RAM), a MRAM (Magnetoresistive RAM), or a combination thereof.

[0019] The storage 203 stores an OS, various programs described later, and various data. The storage 203 is, for example, a flash memory, a HDD (Hard Disk Drive), a SSD (Solid State Drive), a SCM (Storage Class Memories), or a combination thereof.

[0020] The communication I / F 204 is an interface for connecting the server 1 to external devices including the user terminal 2 and the imaging device 16 via the network N and controlling communication. The communication I / F 204 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited thereto.

[0021] (Hardware Configuration - User Terminal) Figure 3 is a block diagram showing the hardware configuration of user terminal 2. User terminal 2 comprises a processor 201, memory 202, storage 203, communication interface 204, input / output interface 205, input device 206, and output device 207, all of which are connected to each other via bus B for communication. Communication interface 204 is an interface for connecting user terminal 2 to an external device, including server 1, via network N, and for controlling communication. Communication interface 204 is, for example, an adapter compliant with Bluetooth®, Wi-Fi®, ZigBee®, Ethernet®, or optical communication (e.g., Fibre Channel).

[0022] The input / output interface 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, keyboard, touch panel, microphone, scanner, camera, various sensors, operation buttons, or a combination thereof. The output device 207, as a user interface, is, for example, a display, projector, printer, speaker, vibrator, or a combination thereof. In this embodiment, as an example, the output device 207 and the input device 206 are configured as an integrated touch panel display.

[0023] In this embodiment, the program may be written to memory 202 or storage 203 during the manufacturing stage of server 1, or it may be provided to server 1 via network N. It may also be provided to server 1 via a non-temporary, computer-readable recording medium such as disk media (not shown).

[0024] (Functional Configuration - Server) Next, the functional configuration of Server 1 will be described. Figure 4 shows an example of the functional configuration of Server 1. When executing various programs, Server 1 uses the above hardware resources to realize various functions. Server 1 has a communication unit 11, a storage unit 12, and a control unit 13 as the functional configuration that Server 1 realizes. Each functional configuration is realized when the processor 101 reads and executes a program 121 stored in memory 102 or storage 103.

[0025] The control unit 13 includes an image acquisition unit 131 that acquires images captured by the user terminal 2, a division unit 132 that divides the captured images of the item acquired by the image acquisition unit 131 into individual parts of the item, and a detection unit 133 that detects scratches, etc., on the item from the captured images acquired by the image acquisition unit 131. Furthermore, the control unit 13 includes a superposition unit 134 that superimposes the scratches, etc., detected by the detection unit 133 onto each of the parts divided by the division unit 132, and a determination unit 135 that determines the overall value of the item and the value of each individual part from the superimposed images created by the superposition unit 134. Details of these will be described later.

[0026] (Functional Configuration - User Terminal) Next, the functional configuration of the user terminal 2 will be described. Figure 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 uses the above hardware resources to realize various functions. The functional configuration realized by the user terminal 2 includes a communication unit 21, a storage unit 22 in which the program 221 is stored, and a control unit 23. Each functional configuration is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server 1, and a display unit 232 that displays the acquired information on the output device 207.

[0027] Next, the processing flow of the information processing system according to this embodiment will be explained using Figures 6 to 8. Figure 6 is a diagram showing the processing performed by the information processing system 10 according to this embodiment. Figure 7 is a schematic diagram showing an example of a screen displayed on the user terminal 2. Figure 8 is a schematic diagram showing the state of an item divided into its parts.

[0028] First, in step S101, the user terminal 2 transmits the captured image of the item to the server 1, and the image acquisition unit 131 acquires the captured image. Next, in step S102, the server 1 determines whether or not the entire item has been photographed. In this embodiment, it is necessary to photograph the entire item. That is, the user terminal 2 photographs the item from all directions to acquire a captured image of the entire item. In this embodiment, an automobile is used as an example of an item, and images are acquired from eight directions: front, back, left, right, left-right, left-right diagonal front, left-right diagonal front, left-right diagonal rear, and right-right diagonal rear of the automobile. Note that for other items, images may also be acquired from other directions such as the top and bottom surfaces in addition to the front, back, left, right, and diagonal, as needed.

[0029] Here, the process of photographing an object will be explained using Figure 7. Figure 7 is an example of a screen displayed on the display unit of the user terminal 2 when photographing an object. Screen IM1 displays eight screens T1 to T8 necessary for photography. Screens T1 to T8 display the completed photograph of the object and correspond to the shooting directions t1 to t8 shown on screen IM2. Before photographing an object, nothing is displayed on screens T1 to T8, and the shooting directions t1 to t8 on screen IM2 are also blank. In this embodiment, the object is an automobile, so the shooting directions are t1 to t8, but if the object is something other than an automobile, the object to be photographed on screen IM2 will be different, and the shooting directions t1 to t8 will also be different depending on the object.

[0030] Furthermore, the shooting directions t1 to t8 also function as a guide for shooting, not only indicating the direction from which to shoot, but also being blacked out if a shot has already been taken, making it easy to see at a glance whether a shot has been taken or not. Specifically, screen T1 corresponds to shooting direction t1, and since a shot has been taken, the captured image is displayed on screen T1, and shooting direction t1 is blacked out. In contrast, screen T2 is before shooting, and the corresponding shooting direction t2 is white. When the captured image is displayed on all screens T1 to T8 and shooting directions t1 to t8 are blacked out, shooting is complete, and the shooting is completed by pressing the complete button IM3.

[0031] In this embodiment, a schematic diagram of the target item and the shooting directions t1 to t8 are displayed on screen IM2 as a guide when photographing an item. However, this is not limited to this, and other methods such as displaying text or displaying the guide on the shooting screen may also be used. Specifically, when the user terminal 2 starts photographing an item, the shooting direction etc. may be displayed on the screen, and when one photograph is completed, instructions for the next photograph may be displayed. In addition to screen displays, audio guidance may also be provided as a guide during shooting. Furthermore, in this embodiment, already photographed items are blacked out, but this is not limited to this, and other colors such as black may be used, as long as the display allows for the determination of whether or not an item has been photographed.

[0032] Next, in step S102, once all shooting is complete, that is, when the captured images are displayed on all screens T1 to T8 and the shooting directions t1 to t8 are all blacked out, and button IM3 is pressed, the process proceeds to step S103. Next, in step S103, the division unit 132 divides the article into parts from the captured image acquired in step S101. Here, Figure 8 shows an example of an image of an article divided into parts. Figure 8 is an example of a divided front image of the automobile 3, which is the article of this embodiment. As shown in Figure 8, the automobile 3 is divided into parts M1 to M8. In this embodiment, it is divided into parts M1 to M8, but for example, the parts to be divided may be selected according to the article. In the automobile 3 of this embodiment, the front is divided into the windshield, side mirrors, headlights, front grille, and tires. Although not shown, the automobile 3 is divided into side mirrors, doors, windows, door handles, and tires on the side, and into rear glass, taillights, rear bumper, and tailgate on the rear.

[0033] The division of parts is not limited to this and may be modified as appropriate. Furthermore, if the item is not an automobile, it will be divided into parts according to the item. In this embodiment, the division of an item is performed based on a learned model of the parts of the item. In the case of automobile 3 in this embodiment, the captured image is divided into parts M1 to M8 based on the learned model of the automobile. Alternatively, feature points may be extracted from the captured image and the parts divided based on these feature points.

[0034] Next, in step S104, the detection unit 133 detects scratches and other damage to an article from the captured image acquired in step S101. In this embodiment, scratches and other damage are detected based on a learned model of scratches and other damage. In this embodiment, scratches and other damage to an automobile will be described as an example of an article. Scratches and other damage to an automobile include, for example, scratches, abrasions, and dents, and the detection unit 133 detects these scratches and other damage from a learned model of scratches. In the case of other articles, scratches and other damage to articles include defects, damage, and dents. Since scratches and other damage to articles are detected from a learned model of scratches, it is not necessary to acquire an image of a genuine article in advance. Therefore, scratches can be detected regardless of the type of vehicle. In other words, when detecting scratches and other damage by comparing with an image of a genuine article, an image of the genuine article is required in advance, and it is difficult to detect scratches and other damage without an image of the genuine article. However, by detecting scratches and other damage from a learned model of scratches, it is possible to detect scratches and other damage to any article.

[0035] Next, in step S105, the overlay unit 134 overlays the parts divided in step S103 with the scratches, etc., detected in step S104, and outputs the overlaid image in step S106. Specifically, in this embodiment, as shown in part M3 of Figure 8, the detected scratches and parts are overlaid and displayed in a way that highlights the parts in which scratches, etc., were detected. However, the display is not limited to this, and the display may be changed to make it easier to understand, such as by hatching the parts in which scratches, etc., were detected. In addition, in this embodiment, only the front image is shown, but the overlaid image is displayed similarly for images of the sides and back, etc. As a result, scratches, etc., on an item can be detected without preparing a regular image of the item in advance. Note that the order of steps S103 and S104 may be reversed, and the processing may be carried out in parallel. Also, in this embodiment, in step S106, the result of the overlay in step S105 is displayed, but in addition to displaying it, these may be saved on a recording medium such as a database and saved as the state of the item at the time of shooting.

[0036] Figure 9 shows a process that includes a step of determining the overall value of an item and the value of each individual part. The process is the same as in Figure 6, and only the differences will be explained. After superimposing in step S105, in step S107, the determination unit 135 determines the overall value of the item and the value of each individual part. By determining the overall value of the item and the value of each individual part, it is possible to determine whether the item should be sold as a whole or disassembled into individual parts and sold separately in the used goods market. Specifically, the determination unit 135 can determine which parts have scratches, etc., in the superimposed images in step S105, and thus determine the costs required for repairs due to the scratches, etc. In other words, in the automobile 3 of this embodiment, if there is a scratch on part M3 as shown in Figure 8, for example, it can be determined that there is a scratch on the side mirror, and therefore the cost required to repair the scratch on the side mirror can be determined.

[0037] Furthermore, even if damage or other defects have been repaired before the time of shooting, the determination unit 135 determines the value from an external database of repair history. For determining the value of the entire item, for example, by linking with an external database, the vehicle type can be identified and its value in the used car market can be determined. Similarly, by linking with an external database, the used car market value of each part can be determined. In this way, the determination unit 135 can determine the value of the entire item (an automobile in this embodiment) and the value of each part based on the condition of damage, parts, vehicle type, past repair history, etc., using the captured image acquired in step S101 and linking with an external database. Then, in step S106, not only the superimposed image but also the value determined in step S107 is output together. In addition to outputting as described above, if the condition of the item is to be saved, the condition of the item can be saved chronologically, and changes in the market value of the item can also be understood.

[0038] As described above, this embodiment allows for the detection of scratches and other defects on an item without the need to prepare a standard image of the item in advance. Furthermore, since scratches and other defects can be detected on each component, it is possible to evaluate the value of the item as a whole and the value of each component. In addition, by linking with an external database, a clearer value evaluation becomes possible. In short, regardless of the type of item, the value of the item can be automatically evaluated simply by photographing the item itself.

[0039] Furthermore, although an automobile was described as an example of an item in this embodiment, the method is not limited to automobiles and can similarly detect scratches and other damage to individual parts and assess the value of other items. In the case of an automobile, as in this embodiment, the used car market involves both the sale of the entire vehicle and the sale of individual parts. Therefore, it is difficult to know in advance whether it is more appropriate to sell the vehicle as a whole or to dismantle it into individual parts, depending on the vehicle model. However, according to this embodiment, regardless of the vehicle model, the value of the vehicle and the value of each individual part can be determined in advance by photographing the automobile on the spot. In addition, during auctions between consumers, both sellers and buyers can conduct transactions with peace of mind by photographing the automobile, detecting scratches and other damage, and determining its value.

[0040] Although one embodiment of the present invention has been described above, the present invention is not limited to the embodiments described above, and any modifications, improvements, etc. that can achieve the objectives of the present invention are included in the present invention.

[0041] Furthermore, the series of processes described above can be executed by hardware or by software. In other words, the functional configuration is merely illustrative and not particularly limited. That is, it is sufficient for the information processing system to have the functionality to execute the series of processes described above as a whole, and there is no particular limitation on what functional blocks are used to realize this functionality. Also, the location of the functional blocks is not particularly limited and can be arbitrary. For example, the functional blocks of a server may be delegated to a user terminal, etc. Conversely, the functional blocks of a user terminal may be delegated to a server, etc. Moreover, a single functional block may be composed of hardware alone, software alone, or a combination of both.

[0042] For example, when a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0043] Furthermore, for example, a recording medium containing such a program may consist not only of removable media (not shown) distributed separately from the main unit of the device to provide the program to the user, but also of a recording medium provided to the user in a state where it is pre-installed in the main unit of the device.

[0044] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually. Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc. [Explanation of Symbols]

[0045] 10. Information Processing Systems 2. User terminal (information processing device) 1 Server (Information Processing Unit) 131 Image acquisition unit 132 divisions 133 Detection Unit 134 Overlapping section

Claims

1. An information processing method performed by an information processing device, The image acquisition step involves obtaining the captured image, A division step in which the captured image is divided into parts based on a learned parts model, A detection step of detecting scratches on the article from the captured image based on a trained scratch model, A superimposing step in which the parts divided in the division step and the scratches detected in the detection step are superimposed, A determination step in which the overall value of the article and the value of each part of the article are determined from the images of each part superimposed in the superimposition step, Includes, In the aforementioned determination step, the system refers to a database of the repair history of the item in the used goods market to determine whether to sell the item as a whole or to sell it disassembled into individual parts. Information processing methods.

2. A guide step that transmits information to guide the shooting direction so that a complete image of the article can be obtained when acquiring the aforementioned captured image. Includes The information processing method according to claim 1.

3. The article is an automobile, In the determination step, the vehicle type is identified, the value of the identified vehicle type in the used car market is determined, and based on the cost required to repair the detected damage, a decision is made on whether to sell the vehicle as a whole or to disassemble it and sell the parts separately. The information processing method according to claim 1.

4. The aforementioned damage includes at least a loss, damage, or dent in the article. The information processing method according to any one of claims 1 to 3.

5. On the computer, The image acquisition step involves obtaining the captured image, A division step in which the captured image is divided into parts based on a learned parts model, A detection step of detecting scratches on the article from the captured image based on a trained scratch model, A superimposing step in which the parts divided in the division step and the scratches detected in the detection step are superimposed, A determination step in which the overall value of the article and the value of each part of the article are determined from the images of each part superimposed in the superimposition step, Includes, In the aforementioned determination step, the system refers to a database of the repair history of the item in the used goods market to determine whether to sell the item as a whole or to sell it disassembled into individual parts. A program for executing information processing methods.

6. An information processing device comprising an information processing system, An image acquisition unit that acquires captured images, A division unit that divides the captured image into individual parts based on a learned parts model, A detection unit that detects scratches on the article from the captured image based on a trained scratch model, An overlapping section in which the parts separated by the aforementioned dividing section and the scratches detected by the aforementioned detection section are superimposed, A determination unit determines the overall value of the article and the value of each part of the article from the images of each part superimposed in the superimposing unit, Includes, The determination unit, in the used goods market, refers to a database of the repair history of the item to determine whether to sell the item as a whole or to sell it after disassembling it into individual parts. Information processing device.

Images