Image processing system and image processing method

Abstract

To make it possible to achieve selection of an area surrounding an article such as a component and a tool in the case of performing work analysis without designating it each time a product is changed.SOLUTION: An image processing system includes an acquisition unit, a determination unit, and a notification unit. The acquisition unit acquires a photographed image obtained by photographing a work area in a production line. The determination unit automatically selects, from the photographed image, an area in which an article used for production of a product in the work area is arranged, and determines whether the article is appropriately arranged in the area. The notification unit notifies information corresponding to determination results by the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system and an image processing method. [Background technology]

[0002] Technology has been disclosed regarding a cloud AI image analysis service using camera-equipped LED lighting. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] “ViewLED Series | Camera-equipped LED lighting ViewLED / Cloud AI image analysis service ViewLED Solution | Product introduction | Toshiba Lighting & Technology Corporation”, [online], [searched on September 12, 2023], Internet<https: / / www.tlt.co.jp / tlt / products / viewled / viewled.htm> Summary of the Invention [Problem to be solved by the invention]

[0004] However, when performing work analysis, which is one of the cloud AI image analysis services that uses camera-equipped LED lighting, a person must visually specify from the image (video) where parts, tools, and other items placed on the workbench are located and surround it with an area.However, on manual production lines in factories, low-volume, high-mix production is assumed, so when performing work analysis, it is necessary to specify the work analysis area every time the product being manufactured is changed, which is a time-consuming task.

[0005] The present application has been made in consideration of the above, and aims to provide an image processing system and an image processing method that make it possible to select an area surrounding an item such as a part or tool when performing work analysis, without having to specify the area each time a product changes. [Means for solving the problem]

[0006] The image processing system according to the present application includes an acquisition unit, a determination unit, and a notification unit. The acquisition unit acquires a captured image of a work area in a production line. The determination unit automatically selects an area in the work area where an item to be used in the production of a product is placed from the captured image, and determines whether the item is appropriately placed in the area. The notification unit notifies information according to the determination result by the determination unit. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the analysis server according to the embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating the flow of processing in the entire image processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an area on a workbench and an image of an item corresponding to the area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The image processing system 1 according to the embodiment described below includes an acquisition unit 331, a determination unit 332, and a notification unit 335. The acquisition unit 331 acquires a captured image of a work area (work bench) in a production line. The determination unit 332 automatically selects an area in the work area where items (parts, tools, etc.) used in the production of a product are placed from the captured image, and determines whether the items are appropriately placed in the area. The notification unit 335 notifies information according to the determination result by the determination unit 332.

[0009] Moreover, the image processing system 1 according to the embodiment described below further includes a setting unit 333. The setting unit 333 receives settings of an area in the work area where items used in the production of the product are placed and information about items corresponding to the area (area-corresponding items) at the time of the initial production.

[0010] Furthermore, in the image processing system 1 according to the embodiment described below, the determination unit 332 automatically selects, for each item, an area in which the item is placed, and determines whether the item is appropriately placed in that area.

[0011] In the image processing system 1 according to the embodiment described below, the determination unit 332 identifies the area and the item based on the production schedule.

[0012] In addition, in the image processing system 1 according to the embodiment described below, the notification unit 335 notifies a worker in the work area of ​​where and what items to place in the work area for use in the production of a product.

[0013] Hereinafter, an image processing system according to an embodiment will be described with reference to the drawings. In the embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0014] FIG. 1 is a diagram showing a configuration example of an image processing system according to an embodiment. As shown in FIG. 1, the image processing system 1 includes a lighting fixture 10, a camera 11, a repeater 20, an analysis server 30, and a work terminal 40. Here, the lighting fixture 10 and the camera 11 are integrated as a camera-equipped LED lighting. The lighting fixture 10 includes lighting fixtures 10a, 10b, and 10c. Similarly, the camera 11 includes cameras 11a, 11b, and 11c. In this embodiment, the camera 11 captures the workbench WB to be captured and objects (objects) such as a part A and a tool T (or equipment) placed thereon. In addition, a worker W may be present in the vicinity of the workbench WB. The camera 11 may continue capturing images all the time, or may capture images only when instructed to do so. However, these are merely examples.

[0015] Furthermore, the lighting fixtures 10 and the camera 11 are connected to the repeater 20, the repeater 20 is connected to the analysis server 30, and the analysis server 30 is connected to the work terminal 40, and these are connected wirelessly or by wire so as to be capable of communicating directly or indirectly. There may be more than one repeater 20. The image processing system 1 may be configured without the repeater 20 by directly connecting the lighting fixtures 10 and the camera 11 to the analysis server 30. The analysis server 30 is installed on the cloud 2, and is connected to each device via a network N such as the Internet (see FIG. 2). The analysis server 30 may be configured by multiple servers on the cloud 2.

[0016] The work terminal 40 is a terminal device such as a PC (Personal Computer) used by the worker W. The work terminal 40 may also be a smart device such as a smartphone or a tablet terminal. Note that, in reality, the person who operates the work terminal 40 may not be the worker W, but may be a work manager or supervisor, an operator, or the like.

[0017] The configuration of the analysis server 30 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the analysis server according to the embodiment. As shown in Fig. 2, the analysis server 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0018] The communication unit 31 is realized by, for example, a network interface card (NIC), etc. The communication unit 31 is also connected to each device so as to be able to communicate with them via a network N (see FIG. 2) such as the Internet.

[0019] The storage unit 32 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a HDD, an SSD, an optical disk, etc. The storage unit 32 may store captured image data, an acquired work schedule, etc.

[0020] The control unit 33 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., executing various programs stored in a storage device inside the analysis server 30 in a storage area such as a RAM. In the example shown in FIG. 3, the control unit 33 has an acquisition unit 331, a determination unit 332, a setting unit 333, an automatic processing unit 334, a notification unit 335, and an operation analysis unit 336.

[0021] The acquisition unit 331 includes a schedule acquisition unit 331a and an image acquisition unit 331b. The schedule acquisition unit 331a acquires production schedule information created by a work manager or a supervisor at the time of the first production. The image acquisition unit 331b acquires an image (video) of the workbench WB of the production line captured by a camera 11 installed above the workbench WB of the production line. The image may be a video or a still image. At this time, items (objects) such as parts A and tools T (or equipment) are placed on the workbench WB. In addition, a worker W may be present around (near) the workbench WB. The image acquisition unit 331b may send a command for shooting to the camera 11 and acquire an image captured by the camera 11 in response to the command. Alternatively, the image acquisition unit 331b may acquire real-time images from the camera 11 that is constantly shooting.

[0022] The determination unit 332 includes an initial determination unit 332a and an arrangement determination unit 332b. The initial determination unit 332a determines whether or not the product is the first to undergo work analysis. The arrangement determination unit 332b determines whether or not an appropriate item has been placed in an appropriate area. Here, the area refers to an area surrounding items such as part A and tool T when performing work analysis.

[0023] The setting unit 333 includes an image display unit 333a, an information input unit 333b, and an item placement management unit 333c. The image display unit 333a displays the image acquired by the acquisition unit on the display of the work terminal 40 via the communication unit 31. The information input unit 333b accepts the selection (designation) of an area for performing a work analysis performed by the worker W while viewing the image displayed on the display of the work terminal 40, and input of information on an item (area-corresponding item) corresponding to the area as a necessary item for the work analysis. The item placement management unit 333c records and manages information on the position and shape of the area selected by the information input unit 333b and information on the necessary items input by the information input unit 333b. The item placement management unit 333c may construct a model by machine learning. For example, the item placement management unit 333c may input the production schedule information acquired by the schedule acquisition unit 331a, information on the position and shape of the area selected by the information input unit 333b, and information on the necessary items input by the information input unit 333b (information on items corresponding to the area) as learning data, and construct a model that estimates the position and shape of the area and the items corresponding to the area (area-corresponding items). Alternatively, the item placement management unit 333c may make an AI (Artificial Intelligence) such as a GPT (Generative Pretrained Transformer) learn the production schedule information, information on the position and shape of the area, and information on items corresponding to the area.

[0024] The automatic processing unit 334 includes a schedule reading unit 334a and an area selection unit 334b. The schedule reading unit 334a reads production schedule information in the production line. The area selection unit 334b ​​automatically selects an area for performing work analysis based on information managed by the item placement management unit 333c. The area selection unit 334b ​​may input the production schedule information read by the schedule reading unit 334a to the model constructed by the item placement management unit 333c, and estimate the position and shape of the area and the item corresponding to the area (area-corresponding item). Alternatively, the area selection unit 334b ​​may input a prompt (instruction sentence) including the production schedule information to an AI such as GPT, and determine the position and shape of the area and the item corresponding to the area (area-corresponding item) based on a response from the AI.

[0025] The notification unit 335 includes an item placement transmission unit 335a and a warning unit 335b. The item placement transmission unit 335a transmits appropriate item placement to the worker W so that the worker W appropriately (correctly) places items in the area automatically selected by the area selection unit 334b. The warning unit 335b issues a warning when the worker W does not appropriately place items as transmitted by the item placement transmission unit 335a.

[0026] The task analysis unit 336 performs task analysis (or analysis).

[0027] FIG. 3 is a flowchart illustrating the flow of processing in the entire image processing system according to this embodiment.

[0028] First, the schedule acquisition unit 331a acquires production schedule information created by a work manager or a supervisor (step S101). For example, the schedule acquisition unit 331a acquires production schedule information output from a schedule output mechanism. The creation and acquisition of production schedule information is to follow the procedure for creating and acquiring production schedule information in a general production line. The schedule acquisition unit 331a may acquire production schedule information only at the first production (once). However, if the production schedule information is updated externally, the schedule acquisition unit 331a may acquire the updated production schedule information again. The production schedule information includes information such as (1) product type name, (2) assembly time, (3) production lot number, and (4) number of assembly personnel. The production schedule information may be stored in the storage unit 32 or in a server other than the analysis server 30.

[0029] Next, the image acquisition unit 331b acquires an image (video) captured by the camera 11 installed above the workbench WB (step S102). An example of the camera 11 is a base light type of camera-equipped LED lighting (ViewLED). The camera 11 may be mounted on the ceiling or may be one that can be set at an angle. In addition, multiple cameras 11 may capture the same workbench WB (and its surroundings).

[0030] Next, the first-time determination unit 332a determines whether or not the product is one for which work analysis is to be performed for the first time (step S103). In the image processing system 1 according to this embodiment, when work analysis is performed, the subsequent process differs depending on the situation.

[0031] If the product is being subjected to work analysis for the first time (step S103: Yes), the image display unit 333a displays the image (video) acquired by the acquisition unit on the display of the work terminal 40 via the communication unit 31 (step S104).

[0032] Next, when the worker W manually selects (specifies) an area while viewing the image (image acquired by the acquisition unit) displayed on the work terminal 40, the information input unit 333b accepts the selection of the area (step S105). Note that in reality, the person who selects (specifies) the area may not be the worker W, but may be a work manager or supervisor, an operator, or the like.

[0033] Furthermore, when the worker W inputs (1) the product type name and (2) necessary information about items such as part A and tool T corresponding to the previously selected area, the information input unit 333b accepts the input of the necessary information (step S106). Note that in reality, the person who inputs the necessary information may not be the worker W but may be a work manager or supervisor, an operator, etc.

[0034] Thereafter, the item placement management unit 333c records and manages (1) the area information selected by the worker W, and (2) necessary information related to the item (area-corresponding item) corresponding to the area input by the worker W (step S107). That is, the item placement management unit 333c stores and manages this information in the storage unit 20. At this time, the item placement management unit 333c holds the (1) area information and (2) necessary information in a format in which there is one-to-one correspondence.

[0035] The item placement management unit 333c may construct a model by machine learning. For example, the item placement management unit 333c may input the production schedule information acquired by the schedule acquisition unit 331a, information on the position and shape of the area selected by the information input unit 333b, and information on the necessary items input by the information input unit 333b (information on items corresponding to the area) as learning data, and construct a model that estimates the position and shape of the area and the items corresponding to the area (area-corresponding items). Alternatively, the item placement management unit 333c may have an AI (Artificial Intelligence) learn the production schedule information, information on the position and shape of the area, and information on items corresponding to the area.

[0036] The areas on the workbench and the items corresponding to the areas will be described with reference to FIG. 4. FIG. 4 is a diagram showing an image of the areas on the workbench and the items corresponding to the areas. For example, as shown in FIG. 4, when there are three selected areas (R1, R2, R3), the item placement management unit 333c determines items corresponding to the areas (area-corresponding items) such as a tool (T), part 1 (A1), and part 2 (A2) from the rectangle at the bottom of the image for each area. In the example shown in FIG. 4, the item placement management unit 333c determines to place the tool (T) in area 1 (R1), part 1 (A1) in area 2 (R2), and part 2 (A2) in area 3 (R3).

[0037] After the above flow, the work analysis unit 336 performs a work analysis (or analysis) (step S108).

[0038] On the other hand, if the product is not the product for which work analysis is being performed for the first time but is the product for which work analysis has been performed in the past (step S103: No), the schedule reading unit 334a reads production schedule information related to the product (step S109).

[0039] Furthermore, the area selection unit 334b ​​refers to the production schedule information read by the schedule reading unit 334a and the information stored in the memory unit 20 when the work analysis was performed for the first time (information managed by the item placement management unit 333c), and based on this information, determines (1) the area position, (2) the area shape, and (3) the items corresponding to the area for the product type name identified from the production schedule information (step S110).

[0040] The area selection unit 334b ​​may input the production schedule information read by the schedule reading unit 334a to the model constructed by the item placement management unit 333c, and estimate the location and shape of the area and the item corresponding to the area (area-corresponding item). Alternatively, the area selection unit 334b ​​may input a prompt (instruction sentence) including the production schedule information to the AI, and determine the location and shape of the area and the item corresponding to the area (area-corresponding item) based on the response from the AI.

[0041] Next, the area selection unit 334b ​​automatically selects an area for the workbench WB in the image captured by the camera 11 based on the determined information (step S111).

[0042] Next, the item placement transmission unit 335a instructs the worker W by any means so that items such as parts A and tools T corresponding to the selected area are placed appropriately (correctly) (step S112). Note that in the initial state before instructing the worker W, the placement determination unit 332b may determine whether the items are placed appropriately in the selected area, and if the items are not placed appropriately, the warning unit 335b may instruct the worker W to place the items appropriately.

[0043] As a method (any means) of giving instructions to the worker W, for example, it is assumed that an image (video) actually captured by a camera and an image including information on the selected area and the items corresponding to that area, as shown in FIG. 4, are output to a display of a PC or the like. In addition to a display, it is also conceivable to print out the image so that it can be easily confirmed at the workbench. In addition, it is also possible to refer to it by transferring the data to a smartphone or tablet terminal and displaying it. It is also conceivable to transmit instructions via a display, speaker, etc. installed at the work site.

[0044] After that, the placement determination unit 332b determines whether the worker W was able to properly place the appropriate item corresponding to the product model name in the appropriate area in response to this instruction (step S113). If the similarity (similarity, similarity rate) between the area and image when the product is first made (at the time of initial production) and the area and image this time is equal to or greater than a predetermined value, it is deemed OK (permitted). The item may be recognized (specified, identified) by image recognition or object recognition, or may be recognized by attaching (attaching) a barcode to the surface of the item or to a box (packaging material) containing the item and reading the barcode. The barcode may be a sticker or the like. Also, multiple cameras may be linked together. For example, one camera may recognize items 1 and 2 from above, and another camera may recognize item 3 from the side or diagonally.

[0045] If the items are not properly placed (step S113: No), the warning unit 335b issues an alert to prompt the user to properly place the items again (step S114). As a method of issuing the alert, for example, a warning sound from a speaker linked to the camera or a vibration of a mobile terminal such as a smartphone may be used.

[0046] If the items are / are appropriately arranged (step S113: Yes), the work analysis unit 336 performs a work analysis (or analysis) (step S108).

[0047] As described above, the image processing system 1 according to the embodiment automatically selects an area surrounding a part by utilizing the fact that the parts used in the work are uniquely determined based on the production schedule. The worker W appropriately places items such as parts and tools used during manufacturing in predetermined locations based on a production schedule created in advance. In addition, in order to properly place items, the system transmits the placement positions of items such as parts and tools corresponding to the product to the worker W by any method. Furthermore, if the worker W places items such as parts and tools incorrectly, a warning is issued by any means to prompt the worker to make corrections.

[0048] As a result, it is possible to obtain an effect that it is possible to select an area surrounding an item such as a part or tool when performing an operation analysis without having to specify the area every time a product is changed.

[0049] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment is included in the scope of the invention and its equivalents as well as in the scope and gist of the invention. In addition, this embodiment can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0050] 1. Image Processing System 10, 10a, 10b, 10c lighting fixtures 11, 11a, 11b, 11c Camera 20 Repeater 30 Analysis Server 331 Acquisition Department 332 Judgment section 333 Settings 334 Automatic Processing Unit 335 Information Department 336 Work Analysis Department 40 Work Station

Claims

1. An acquisition unit that acquires an image of a work area in a production line; a determination unit that automatically selects an area in which an item to be used in the production of a product is to be placed in the work area from the captured image, and determines whether or not the item is appropriately placed in the area; a notification unit that notifies information corresponding to a result of the determination by the determination unit; 1. An image processing system comprising:

2. a setting unit that receives settings of an area in which items to be used in the production of the product in the work area are placed and information on the items corresponding to the area at the time of initial production; 2. The image processing system according to claim 1, further comprising:

3. The determination unit automatically selects, for each item, an area in which the item is to be placed, and determines whether the item is appropriately placed in the area.

2. The image processing system according to claim 1.

4. The determination unit identifies the area and the item based on a production schedule.

2. The image processing system according to claim 1.

5. The notification unit notifies a worker in the work area of ​​where and what to place in the work area regarding the items used in the production of the product in the work area.

2. The image processing system according to claim 1.

6. An image processing method executed by an information processing device, comprising: An acquisition step of acquiring an image of a work area in a production line; a determining step of determining, from the captured image, whether or not a placement position of a step product corresponding to a product to be produced in the work area satisfies a predetermined condition; a notification step of notifying information corresponding to a result of the determination step; 13. An image processing method comprising:

Images