Method for generating teaching data, device for generating teaching data

Abstract

The present invention provides a method for generating teaching data, a device for generating teaching data, and an image processing device. The registration of an image used as teaching data is efficiently performed. The server (300) determines an image area (first image area) including an image of a marker (102a) etc. from a captured image corresponding to image data obtained by capturing an image with a camera (200a) etc., and determines an image area (second image area) of a moving body (such as a forklift (100a) etc.) including the marker (102a) etc. The server (300) generates the second image area as teaching data and performs registration. The teaching data is used in subsequent machine learning.

Description

Technical Field

[0001] The present invention relates to a method for generating teaching data, a device for generating teaching data, and an image processing device. Background Art

[0002] In recent years, with the development of machine learning technology, annotation (teaching data of an image) used in the machine learning is generated. For example, as an example of an operation of specifying an image of a specific part of an object used as teaching data, there is an annotation operation of a medical image based on manual operation (for example, Japanese Patent Laid-Open No. 2020-35095).

[0003] However, in the above technology, the selection of an image required for generating teaching data depends mainly on the manual operation of an operator, which is time-consuming. Summary of the Invention

[0004] In a method for generating teaching data according to an embodiment of the present invention, a computer executes: determining a second image region including a first image region for obtaining identification information of a moving body from a captured image, and generating data corresponding to the determined second image region as teaching data for machine learning.

[0005] An advantage of the present invention is that registration of an image used as teaching data can be efficiently performed. Brief Description of the Drawings

[0006] Figure 1 It is a diagram showing an example of the structure of a visible light communication system according to an embodiment of the present invention.

[0007] Figure 2 It is a diagram showing an example of the structure of a forklift according to the embodiment.

[0008] Figure 3 It is a diagram showing an example of the structure of a camera, a server, and a database according to the embodiment.

[0009] Figure 4 It is a diagram showing an example of captured image data according to the embodiment.

[0010] Figure 5A It is a diagram showing an example of an image region of a sign and an image region of a forklift according to the embodiment, Figure 5B is compared with Figure 5A a diagram of a case where the image region of the sign is small.

[0011] Figure 6 It is a diagram showing an example of area data according to the embodiment.

[0012] Figure 7It is a diagram showing an example of teaching data related to this embodiment.

[0013] Figure 8A It is a diagram showing an example of an image area of a forklift related to this embodiment. Figure 8B It is shown in Figure 8A It is a diagram showing an example of an image area of a forklift in which the color of the image area of the marker is changed.

[0014] Figure 9 It is a flowchart showing an example of teaching data generation processing related to this embodiment.

[0015] Figure 10 It is a flowchart showing an example of color change processing of an image area of a marker related to this embodiment.

[0016] Figure 11 It is a flowchart showing an example of teaching data generation processing related to another embodiment of the present invention.

[0017] Figure 12 It is a flowchart showing an example of alarm notification processing related to this embodiment.

[0018] Figure 13A It is a diagram showing another example of an image area of a forklift. Figure 13B It is related to Figure 13A It is a diagram showing a case where the image area of the forklift is smaller compared to

[0019] Figure 14A It is a diagram showing another example of an image area of a forklift. Figure 14B It is related to Figure 14A It is a diagram showing a case where the image area of the marker is located on the image compared to Detailed Embodiment

[0020] The visible light communication system related to the embodiment of the present invention will be described below with reference to the accompanying drawings.

[0021] Figure 1 It is a diagram showing an example of the structure of a visible light communication system. As Figure 1 shown, in the space S where the visible light communication system 1 is operated, shelves 400a and 400b are provided, including forklifts 100a and 100b (hereinafter, when not specifically limiting forklifts 100a and 100b, they are appropriately referred to as "forklift 100" or "mobile body"), cameras 200a, 200b, 200c, and 200d (hereinafter, when not specifically limiting cameras 200a, 200b, 200c, and 200d, they are appropriately referred to as "camera 200"), hub 210, server 300, and database 500.

[0022] The forklift 100a includes a sign (light-emitting body) 102a of an LED (Light Emitting Diode), and the forklift 100b includes a sign 102b (hereinafter, appropriately referred to as "sign 102" without individually limiting the signs 102a and 102b). The server 300 is connected to the camera 200 via the hub 210. In addition, it is connected to the database 500 via a network LAN (Local Area Network) not shown in the figure.

[0023] In the present embodiment, the sign 102 installed on the forklift 100 causes the emission color to change in time series corresponding to the communication data including the identification information of the forklift 100 as the transmission target, and transmits it through visible light communication. In the present embodiment, the identification information is a classification ID indicating that the forklift 100 is a forklift. In addition, the identification information may include a vehicle number or the like that uniquely identifies the information of the forklift 100 in addition to the classification ID.

[0024] On the other hand, the camera 200 captures an image of the entire space S. The server 300 obtains the position (two-dimensional position) of the sign 102 in the image and the position (three-dimensional position) of the sign 102 in the space S from the image of the entire space S obtained by the imaging of the camera 200 through visible light communication, and further demodulates the content of the light emission that changes in time series of the sign 102, thereby obtaining communication data from the forklift 100. In addition, in the present embodiment, the server 300 generates teaching data used for identifying the image area of the forklift 100 in the image in machine learning.

[0025] Figure 2 It is a diagram showing an example of the structure of the forklift 100. As Figure 2 shown, the forklift 100 includes a sign 102, a control unit 103, a memory 104, a communication unit 110, a drive unit 112, and a battery 150.

[0026] The control unit 103 is constituted by, for example, a CPU (Central Processing Unit). The control unit 103 controls various functions of the forklift 100 by executing software processing in accordance with a program stored in the memory 104.

[0027] The memory 104 is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The memory 104 stores various information (programs, etc.) for control and the like used in the forklift 100.

[0028] The communication unit 110 is, for example, a LAN card. The communication unit 110 performs wireless communication with the server 300 and the like. The battery 150 supplies power required for the operation of each part of the forklift 100.

[0029] The control unit 103 reads out the identification information of the forklift 100 stored in the memory 104.

[0030] A light emission control unit 124 is formed within the control unit 103. The light emission control unit 124 determines a light emission pattern in which the light emission color changes in time series corresponding to the identification information as communication data.

[0031] Furthermore, the light emission control unit 124 outputs the information of the light emission pattern to the drive unit 112. The drive unit 112 generates a drive signal for changing the hue of the light emitted from the sign 102 in time corresponding to the information of the light emission pattern from the light emission control unit 124. The sign 102 emits light with a hue that changes in time corresponding to the drive signal output from the drive unit 112. For example, the light emission color is one of the three primary colors, and is a color in the wavelength band used in color modulation in visible light communication, namely red (R), green (G), or blue (B).

[0032] Figure 3 It is a diagram showing an example of the structures of the camera 200, the server 300, and the database 500. As Figure 3 shown, the camera 200 and the server 300 are connected via a hub 210, and are connected to the database 500 via a network LAN. The camera 200 includes an imaging unit 202 and a lens 203. The server 300 includes a control unit 302, an image processing unit 304, a memory 305, an operation unit 306, a display unit 307, and a communication unit 308. The database 500 has an imaging data storage unit 501, an area data storage unit 502, and a teaching data storage unit 503.

[0033] The lens 203 in the camera 200 is composed of a zoom lens or the like. The lens 203 moves by a zoom control operation from the operation unit 306 in the server 300 and a focus control performed by the control unit 302. The imaging angle of view and the optical image captured by the imaging unit 202 are controlled by the movement of the lens 203.

[0034] The imaging unit 202 has a light-receiving surface including an imaging surface, which is composed of a plurality of light-receiving elements regularly arranged two-dimensionally. The light-receiving elements are, for example, imaging devices such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). Based on a control signal from the control unit 302 in the server 300, the imaging unit 202 images (receives light) an optical image incident through the lens 203 at a given range of imaging angles, converts the image signal within this imaging angle into digital data to generate a frame. In addition, the imaging unit 202 continuously performs imaging and frame generation in time, and outputs the digital data of consecutive frames to the image processing unit 304.

[0035] The image processing unit 304 corrects distortion, adjusts color matching, and removes noise from the digital data of the frames output from the imaging unit 202 based on a control signal from the control unit 302, and outputs the result to the control unit 302.

[0036] The control unit 302 is composed of a processor such as a CPU, for example. By executing software processing in accordance with a program stored in the memory 305, the control unit 302 controls various functions of the server 300, such as the Figures 9 to 12 processing shown below.

[0037] The memory 305 is, for example, RAM or ROM. The memory 305 stores various information (programs, etc.) used for control and the like in the server 300.

[0038] The operation unit 306 is composed of a numeric keypad, function keys, etc., and is an interface for inputting the operation content of the user. The display unit 307 is composed of, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an EL (Electro Luminescence) display, etc. The display unit 307 displays an image according to the image signal output from the control unit 302. The communication unit 308 is, for example, a LAN card. The communication unit 308 communicates with an external communication device.

[0039] In the control unit 302, a registration unit 332, an image area range determination unit 334, a movement detection unit 336, a color change unit 338, an image area comparison unit 340, and a notification unit 342 are formed.

[0040] The registration unit 332 attaches the identification information of the image data, that is, the image ID, to the digital data (image data) of a plurality of frames output from the imaging unit 202 in the camera 200, to generate imaging image data.

[0041] Figure 4 This is a diagram showing an example of captured image data. The image ID is composed of the camera 200 that outputs the corresponding image data, in other words, the identification information of the camera that captured the corresponding image, i.e., the camera ID, and the capture date and time of the camera 200.

[0042] In addition, each piece of information of the image ID, the camera ID, and the capture date and time, together with the image data of the captured image data, is stored as personal profile data, but it can also be set as independent data corresponding to the image data.

[0043] Returning again to Figure 3 for explanation. The registration unit 332 registers the generated captured image data into the captured image data storage unit 501 in the database 500.

[0044] The image area range determination unit 334 obtains the luminance values of the respective pixels constituting the frame for the digital data of a plurality of frames output by the imaging unit 202. Next, the image area range determination unit 334 regards the positions of the pixels with luminance values equal to or higher than a given value in the frame as the positions of the marker 102. Furthermore, the image area range determination unit 334 performs decoding processing on the change in the emission color at the position of the marker 102 within the frame, and obtains the classification ID included in the communication data transmitted by the marker 102. The following processing is performed separately for each classification ID.

[0045] The image area range determination unit 334 reads the captured image data from the captured image data storage unit 501, and determines whether the image area of the marker 102 is included in the captured image corresponding to the image data included in the captured image data. Specifically, when there are pixels with luminance values equal to or higher than a given value in the captured image, the image area range determination unit 334 determines that the image area of the marker 102 is included.

[0046] In the case where the image area of the marker 102 is included, the image area range determination unit 334 determines the range of the image area of the image of the forklift 100 that is determined to exist in the captured image corresponding to the image data included in the captured image data. Specifically, the image area range determination unit 334 sets a pre-set range as the range of the image area of the forklift 100 such that, with the position of the image area of the marker 102 in the captured image as the center, in the vertical direction, the image of the marker 102 is slightly above, and in the horizontal direction, the image of the marker 102 reaches the center. In addition, the size is increased in proportion to the size of the image area of the marker 102, and the image of the forklift 100 is substantially enclosed within this range.

[0047] At this time, the image area range determination unit 334 discriminates the size of the image area of the marker 102. The image area range determination unit 334 determines the range of the image area of the forklift 100 such that the larger the image area of the marker 102, the larger the image area of the forklift 100 in the captured image. In addition, the image area of the forklift 100 in the captured image is determined without performing detection processing or image recognition on the image of the forklift 100 from the captured image. That is, based on the position and size of the marker 102 in the captured image, the position and range of the image of the forklift 100 having the marker 102 in the captured image are estimated, and the determination is made based on the estimation result.

[0048] Figure 5A And Figure 5B is a diagram showing an example of the image area of the marker 102 and the image area of the forklift 100. If comparing Figure 5A the captured image 600a and Figure 5B the captured image 600b, Figure 5A the image area 602a (first image area) of the marker 102a in Figure 5B is larger than the image area 602b (first image area) of the marker 102a in Figure 5A For this reason, Figure 5B the image area 604a (second image area) of the forklift 100a in

[0049] The identification information of the moving body can be obtained from the first image area. The first image area is included in the second image area. In one embodiment of the present invention, the processor 302 determines the second image area from the captured image.

[0050] In this way, the image area range determination unit 334 performs control so that the size of the determined image area changes in proportion to the size of the image of the identification information, that is, the marker 102.

[0051] That is, the processor 302 determines the size of the second image area based on the size of the first image area.

[0052] Returning to Figure 3 for explanation again. After determining the range of the image area of the forklift 100, the image area range determination unit 334 generates area data, which is information for determining the image area of the forklift 100.

[0053] Figure 6 is a diagram showing an example of the area data. Figure 6The area data 512 shown contains: an image ID of a captured image including an image area corresponding to the forklift 100; and image area data for determining the image area of the forklift 100. The image area data represents the coordinates of the upper left corner of the image area of the forklift 100 in the captured image, the length in the horizontal direction, i.e., the X direction, and the length in the vertical direction, i.e., the Y direction. Additionally, Figure 6 The image area data shown represents an example of the case where the image area of the forklift 100 is rectangular. The form of the image area data differs according to the shape of the image area of the forklift 100.

[0054] Returning once again to Figure 3 for explanation. The registration unit 332 registers the area data generated by the image area range determination unit 334 in correspondence with the captured image data in the area data storage unit 502 in the database 500.

[0055] Next, the image area range determination unit 334 generates teaching data corresponding to the generated area data. Figure 7 is a diagram showing an example of the teaching data. As Figure 7 shown, the teaching data 513 consists of a classification ID and image data (forklift area image data) corresponding to the image area of the forklift 100.

[0056] When generating the teaching data, the image area range determination unit 334 obtains the classification ID contained in the communication data obtained through the above processing. Next, the image area range determination unit 334 reads out the captured image data including the image ID within the area data corresponding to the classification ID from the captured image data storage unit 501. Further, the image area range determination unit 334 cuts out the range determined by the image area data in the generated area data from the image data in the read captured image data and attaches it to the classification ID as the forklift area image data.

[0057] Returning once again to Figure 3 for explanation. The registration unit 332 registers the teaching data generated by the image area range determination unit 334 in the teaching data storage unit 503 in the database 500.

[0058] After generating and registering the teaching data, the color change unit 338 in the control unit 302 changes the color of the image area of the marker 102 in the image corresponding to the forklift area image data in the teaching data to the color around the marker 102. The color change unit 338 can, in the same manner as above, determine the pixels with a brightness value equal to or greater than a given value as the image area of the marker 102. By changing the color of the image area of the marker 102, for example Figure 8A the image area 604a of the forklift 100 shown becomes Figure 8BThe image area 614a of the forklift 100 shown. In this way, by eliminating the image of the sign 102, data of the image area of the forklift 100 with high versatility is generated as teaching data.

[0059] The processing performed by the server 300 will be described below with reference to the flowchart.

[0060] Figure 9 It is a flowchart showing an example of the teaching data generation process. The image area range determination unit 334 in the control unit 302 of the server 300 determines whether there is image data for which the teaching data generation process has not been executed registered in the captured image data storage unit 501 (step S101). When there is no image data for which the teaching data generation process has not been executed, that is, when it is determined that teaching data has been generated from all the captured image data registered in the captured image data storage unit 501 (step S101 “No”), the series of operations related to the teaching data generation process ends.

[0061] On the other hand, when it is determined that there is image data for which the teaching data generation process has not been executed registered (step S101 “Yes”), the image area range determination unit 334 reads out the image data for which the teaching data generation process has not been executed from the captured image data storage unit 501 (step S102). Next, the image area range determination unit 334 determines whether the image area (the first image area) of the sign 102 is included in the captured image corresponding to the image data included in the read captured image data (step S103). When it is determined that the image area of the sign 102 is not included in the captured image (step S103 “No”), it is determined that the image area that becomes the object of the teaching data is not included in the captured image data, and the operations after step S101 are repeated again.

[0062] On the other hand, when it is determined that the image area of the sign 102 is included in the captured image (step S103 “Yes”), the image area range determination unit 334 determines the range of the image area (the second image area) of the forklift 100 to be used as teaching data in the captured image (step S104). Next, the image area range determination unit 334 generates area data, which is information on the image area of the forklift 100 determined in the captured image. The registration unit 332 registers the area data extracted and generated by the image area range determination unit 334 from the captured image in the area data storage unit 502 in the database 500 (step S105).

[0063] Next, the image area range determination unit 334 generates teaching data (data corresponding to the second image area) corresponding to the generated area data. The registration unit 332 registers the teaching data generated by the image area range determination unit 334 in the teaching data storage unit 503 in the database 500 (step S106).

[0064] In addition, the captured images are a plurality of images continuously captured in a time series, and the processor 302 detects the movement of the first image area in the plurality of images, and determines the second image area corresponding to the movement of the first image area.

[0065] Figure 10 It is a flowchart showing an example of the color change process of the image area representing the mark. The color change unit 338 in the control unit 302 of the server 300 determines whether teaching data of the image area where the mark exists is registered in the teaching data stored in the teaching data storage unit 503 (step S201). If it is determined that the teaching data of the image area where the mark exists is not registered in the teaching data storage unit 503 (step S201 "No"), a series of operations ends.

[0066] On the other hand, if it is determined that the teaching data of the image area where the mark exists is registered in the teaching data storage unit 503 (step S201 "Yes"), the color change unit 338 changes the color of the image area of the mark 102 to the color around the mark 102 in the image corresponding to the forklift area image data in the teaching data (step S202).

[0067] That is, the processor 302 changes the color within the first image area within the second image area, and generates data corresponding to the second image area with the color within the first image area changed as teaching data.

[0068] In this way, in the present embodiment, the server 300 determines the image area of the mark 102 in the captured image corresponding to the image data captured by the camera 200, and determines the position and range of the forklift 100 without performing detection processing and image recognition from the captured image. That is, based on the position and size of the mark 102 in the captured image, the position and range within the captured image of the forklift 100 image having the mark 102 are estimated, and the determination is made based on the estimation result. Further, the server 300 generates information for determining the image area of the forklift 100 as teaching data and registers it. Thereby, the deviation due to the shooting environment and the operator during the generation of teaching data is prevented, and in addition, there is no need to rely on manual operation to select images, and the registration of images used as teaching data can be efficiently performed.

[0069] In addition, the server 300 determines the range of the image area of the forklift 100 such that the larger the image area of the marker 102, the larger the image area of the forklift 100. Thus, the situation where the larger the image area of the marker 102, the larger the image area of the forklift 100 can be regarded as being utilized to accurately determine the range of the image area of the forklift 100.

[0070] In addition, the server 300 changes the color of the image area of the marker 102 to the color around the marker 102 in the image corresponding to the forklift area image data in the teaching data. Thus, considering that the marker 102 is usually not installed on the forklift 100, it is possible to generate highly versatile teaching data simulating the state without the marker 102.

[0071] In addition, the server 300 aggregates and generates image data for each classification ID as teaching data. For this reason, in machine learning, it is possible to easily identify an object by classification ID.

[0072] Next, other embodiments will be described. In the present embodiment, Figure 3 The registration unit 332 in the control unit 302 of the server 300 shown, similar to the above, attaches the identification information of the image data, i.e., the image ID, to the digital data (image data) of a plurality of frames output by the imaging unit 202 in the camera 200 to generate imaging image data, and registers it in the imaging image data storage unit 501 in the database 500.

[0073] The image area range determination unit 334, similar to the above, analyzes the image data corresponding to the imaging images respectively from a plurality of cameras 200, and regards the positions of the pixels with a luminance value equal to or higher than a given value in each image data as the positions of the marker 102. Further, the image area range determination unit 334 performs a decoding process on the change in the emission color at the position of the marker 102 within the frame, and obtains the classification ID included in the communication data transmitted by the marker 102. The following processing is performed for each classification ID.

[0074] Next, the image area range determination unit 334 determines the three-dimensional position in the space S of the forklift 100 based on the imaging image data corresponding to the digital data (image data) of the frames output by the imaging units 202 in at least two cameras 200.

[0075] Specifically, the image area range determination unit 334 reads out the imaging image data corresponding to the same imaging date and time imaged by at least two cameras 200 from the imaging image data storage unit 501. Next, the image area range determination unit 334 analyzes the image data in the read imaging image data, and determines the image data with a luminance value equal to or higher than a given value and indicating the same emission mode as the marker 102.

[0076] Furthermore, the image area range determination unit 334 determines the three-dimensional position of the marker 102 in the space S by using information such as the position (two-dimensional position) of the marker 102 in the image corresponding to the image data in each read camera image data, the installation positions of the respective cameras 200, and the imaging ranges of the respective cameras 200, through, for example, the technique described in Japanese Unexamined Patent Application Publication No. 2020-95005.

[0077] Next, the movement detection unit 336 in the control unit 302 determines the manner of change in the three-dimensional position of the marker 102 in the space S that is continuous in time. For example, based on the determined manner of change, it is determined whether, in the behavior of the forklift 100 equipped with the marker 102, there has been a deviation from the action following a given schedule, such as sudden deceleration or sudden stop. For example, the movement detection unit 336 determines the three-dimensional position of the marker 102 in the space S at a given time period, and when the change in the three-dimensional position suddenly becomes small, it is determined that sudden deceleration or sudden stop has occurred in the behavior of the forklift 100.

[0078] In the case where sudden deceleration or sudden stop has occurred in the behavior of the forklift 100, the movement detection unit 336 determines the time at which the sudden deceleration or sudden stop has occurred. The time at which sudden deceleration or sudden stop has occurred can be determined from the imaging date and time in the corresponding camera image data.

[0079] Next, the image area range determination unit 334 reads out from the camera image data storage unit 501: camera image data including the imaging date and time within a given time period including the time at which sudden deceleration or sudden stop has occurred and used for the determination of the three-dimensional position of the marker 102 in the above-mentioned space S; and camera image data including the same camera ID.

[0080] That is, the processor 302 detects sudden deceleration or sudden stop of the moving body, and generates the teaching data based on the camera image of the moving body corresponding to the time at which the sudden deceleration or sudden stop has occurred.

[0081] Furthermore, the image area range determination unit 334 analyzes the image data in the read camera image data, and determines the image area of the marker 102 as the image data having a luminance value equal to or higher than a given value and indicating the same light emission mode.

[0082] In the case of an image area including the marker 102, the image area range determination unit 334 determines the range of the image area of the forklift 100 used as teaching data in the captured image corresponding to the image data included in the captured image data. Specifically, the image area range determination unit 334, in the same manner as described above, discriminates the size of the image area of the marker 102 in the captured image. Further, the larger the image area of the marker 102 is, the larger the image area of the forklift 100 in the captured image is regarded as being, and the image area range determination unit 334 determines the range of the image area of the forklift 100 such that the larger the image area of the marker 102 is, the larger the image area of the forklift 100 is.

[0083] After determining the range of the image area of the forklift 100, the image area range determination unit 334, in the same manner as described above, generates area data, which is information determining the image area of the forklift 100, within a given time including the time of sudden deceleration or sudden stop. The registration unit 332 registers the area data within the given time including the time of sudden deceleration or sudden stop generated by the image area range determination unit 334 into the area data storage unit 502 in the database 500.

[0084] Next, the image area range determination unit 334, in the same manner as described above, generates teaching data within a given time including the time of sudden deceleration or sudden stop. The registration unit 332 registers the teaching data within the given time including the time of sudden deceleration or sudden stop generated by the image area range determination unit 334 into the teaching data storage unit 503 in the database 500.

[0085] After generating and registering the teaching data, or in parallel with the generation and registration of the teaching data, a process of alarm notification is performed when the forklift 100 suddenly decelerates or suddenly stops.

[0086] Specifically, the image area comparison unit 340 in the control unit 302 acquires and analyzes the image data from the camera 200, and determines the image area of the marker 102 as the pixels with a brightness value equal to or higher than a given value. Next, the image area comparison unit 340 determines the image area around the determined image area of the marker 102.

[0087] Further, the image area comparison unit 340 compares the image of the image area around the determined image area of the marker 102 with the image corresponding to the forklift area image data in the teaching data within a given time including the time of sudden deceleration or sudden stop registered in the teaching data storage unit 503, and determines whether the two images are the same or similar. In the case where the two images are the same or similar, the notification unit 342 in the control unit 302 performs a notification process such as displaying an alarm on the display unit 307.

[0088] That is, the processor 302 obtains identification information of the moving body based on the captured image, and determines whether the movement of the moving body is a given behavior based on the captured image and the teaching data generated based on the captured image corresponding to the time when the moving body decelerated or stopped suddenly in the past. When it is determined that it is the given behavior, the moving body is notified.

[0089] The processing performed by the server 300 will be described below with reference to the flowchart.

[0090] Figure 11 FIG. is a flowchart showing an example of the teaching data generation process in the present embodiment. The image area range determination unit 334 in the control unit 302 of the server 300 determines the three-dimensional position in the space S of the forklift 100 based on the captured image data corresponding to the digital data (image data) of the frames output by the imaging units 202 in at least two cameras 200 (step S301).

[0091] Next, the movement detection unit 336 determines the manner of change in the three-dimensional position in the space S of the markers 102 that are continuous in time series, and determines whether the behavior of the forklift 100 has decelerated or stopped suddenly (step S302). When it is determined that neither sudden deceleration nor sudden stop has occurred (step S302 "No"), the operations after the determination of the three-dimensional position in the space S of the forklift 100 (step S301) are repeated.

[0092] On the other hand, when it is determined that the forklift 100 has decelerated or stopped suddenly (step S302 "Yes"), the movement detection unit 336 determines the time when the sudden deceleration or sudden stop occurred (step S303).

[0093] Next, the image area range determination unit 334 determines the range of the image area of the forklift 100 in the captured image within a given time including the time when the sudden deceleration or sudden stop occurred (step S304).

[0094] Next, the image area range determination unit 334 generates area data for the area in the space S where sudden deceleration or sudden stop has occurred within a given time including the time when the sudden deceleration or sudden stop occurred. The registration unit 332 registers the area data within a given time including the time when the sudden deceleration or sudden stop occurred in the area data storage unit 502 in the database 500 (step S305).

[0095] Next, the image area range determination unit 334 generates teaching data within a given time period that includes the time of sudden deceleration or sudden stop. The registration unit 332 registers the teaching data within the given time period that includes the time of sudden deceleration or sudden stop into the teaching data storage unit 503 in the database 500 (step S306). After that, until the system stops, the operations after the determination of the three-dimensional position in the space S of the forklift 100 (step S301) are repeated.

[0096] Figure 12 It is a flowchart showing an example of the alarm notification process. The image area comparison unit 340 in the control unit 302 acquires image data from the camera 200 (step S401).

[0097] Next, the image area comparison unit 340 determines the image area around the image area of the marker 102 (step S402).

[0098] Furthermore, the image area comparison unit 340 compares the image of the determined image area around the image area of the marker 102 with the image corresponding to the forklift area image data in the teaching data within a given time period that includes the time of sudden deceleration or sudden stop registered in the teaching data storage unit 503, and determines whether the two images are the same or similar (step S403). When it is determined that the two images are neither the same nor similar (step S403 "No"), the operations after the acquisition of the image data (step S401) are repeated.

[0099] On the other hand, when it is determined that the two images are the same or similar (step S403 "Yes"), the notification unit 342 performs a notification process (step S404). After that, until the system stops, the operations after the acquisition of the image data (step S401) are repeated.

[0100] In this way, in the present embodiment, the server 300 detects the movement of the marker 102, and when sudden deceleration or sudden stop of the forklift 100 occurs, determines the range of the image area of the forklift 100 within a given time period that includes the time of sudden deceleration or sudden stop, and generates teaching data. Furthermore, the server 300 compares the new captured image with the image corresponding to the image data in the teaching data within a given time period that includes the time of sudden deceleration or sudden stop, and performs a given notification process when it is determined that they are consistent or similar. Thereby, it is possible to perform notification using teaching data specialized for abnormal situations such as sudden deceleration or sudden stop of the forklift 100.

[0101] In addition, the present invention is not limited by the description and drawings of the above embodiments, and changes and the like can be appropriately added to the above embodiments and drawings.

[0102] In the above-described embodiment, as Figure 5A and Figure 5B shown, the range of the image area of the forklift 100 is determined such that the larger the image area of the marker 102, the larger the image area of the forklift 100. However, the determination of the range of the image area of the forklift 100 is not limited to this.

[0103] For example, it is also possible that, when a plurality of markers 102 are installed on one forklift 100 at a predetermined interval, the longer the distance between the markers 102 in the image, the closer the forklift 100 is regarded as being to the camera 200, thereby increasing the range of the image area of the forklift 100.

[0104] For example, for Figure 13A and Figure 13B explanation, it is assumed that two markers 102a and 102c are installed on the forklift 100 at a distance L (not shown). In Figure 13A the captured image 600c, since the distance between the markers 102a and 102c in the image is L1, the image area 621a of the forklift 100 is set. In comparison, in Figure 13B the captured image 600d, since the distance between the markers 102a and 102c in the image is L2, which is shorter than the Figure 13A L1 distance, the image area 621b of the forklift 100 is set smaller than the image area 621a.

[0105] That is, when there are a plurality of first image areas included in the captured image, the processor 302 determines the second image area based on the distance between the first image areas.

[0106] In addition, for example, since the camera 200 is usually installed at a high position, considering that the imaging direction is downward, the lower the position of the marker 102 in the image, the closer the forklift 100 is regarded as being to the camera 200, thereby increasing the range of the image area of the forklift 100.

[0107] For example, in Figure 14A the captured image 600e, the image area 631a of the forklift 100 is set based on the position of the marker 102a in the image. In comparison, in Figure 14B the captured image 600d, since the position of the image of the marker 102a is higher than the position of the image of the marker 102a in Figure 14A , the image area 631b of the forklift 100 is set smaller than the image area 631a.

[0108] That is, the processor 302 determines the second image area based on the position of the first image area in the captured image.

[0109] In addition, in the above-described embodiment, the classification ID is set to information indicating a forklift, but it is not limited thereto. In the classification ID, it may also be information for determining the manufacturer, the presence or absence of goods, and information about the goods. In this case, processing is performed for each classification ID, and teaching data is generated for each classification ID.

[0110] In addition, the captured image data and the area data in the above-described embodiment may be associated with each other as teaching data.

[0111] In addition, after teaching data is generated in advance when the forklift 100 suddenly decelerates or stops, when the forklift 100 suddenly decelerates or stops again, notification processing is performed if it is the same as or similar to the teaching data. However, it is not limited thereto, and it is also possible to generate changes in the behavior of the forklift that do not normally occur at this position, such as sudden start, sudden acceleration, and sharp turn, as teaching data in the form of a three-dimensional position. In addition, it is also possible to set it to indicate an abnormality and perform notification processing when it is the same as or similar to the captured image of the forklift 100 later.

[0112] In addition, in the above-described embodiment, it has been described that visible light of red, green, and blue is used for communication, but other colors of visible light may also be used. In addition, the present invention can also be applied to visible light communication in which information is modulated only by changes in the time direction of brightness.

[0113] In addition, the structure of the identification information used to determine the range of the image area of the forklift 100 is not limited to the marker 102. For example, a light source may be formed as a part of an LCD, PDP, EL display, etc. that constitute a display device. Furthermore, instead of the marker 102, an object (paper medium, label, panel, etc.) that determines the range of the image area of the forklift 100 by color, shape, or a geometric pattern such as a barcode may be provided at a position on the forklift 100 that can be visually recognized / photographed by the camera 200 (e.g., the top, side part).

[0114] In addition, the server 300 may incorporate the camera 200.

[0115] In addition, in the above-described embodiment, the program to be executed is stored in a computer-readable recording medium such as a removable hard disk, floppy disk, CD-ROM (Compact Disc - Read Only Memory), DVD (Digital Versatile Disc), MO (Magneto - Optical disc), etc., distributed, and the program is installed, thereby constituting a system that executes the above-described processing.

[0116] Alternatively, the program can also be stored in a disk device or the like of a given server on a network such as the Internet, and for example, downloaded by being superimposed on a carrier wave, etc.

[0117] In addition, in the case where the above functions are implemented by being shared by an OS (Operating System), or in the case where they are implemented through the cooperation between the OS and an application, etc., only the part other than the OS can be stored in a medium for distribution, and also, downloading, etc. can be performed.

[0118] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the relevant specific embodiments, and in the present invention, it includes the invention described in the claims and its equivalent scope.

Claims

1. A method for generating teaching data, characterized in that: It is executed by a computer: A determination process for determining a second image area including a first image area for obtaining identification information of a moving body from a captured image; and A generation process for generating data corresponding to the second image area determined in the determination process as teaching data for machine learning, The captured image is a plurality of images continuously captured in a time series, In the determination process, the size of the second image area is determined based on the size of the first image area, and the movement of the first image area in the plurality of images is detected, and the second image area is determined corresponding to the movement of the first image area.

2. The method for generating teaching data according to claim 1, characterized in that: A change process for changing the color within the first image area is executed, In the generation process, data corresponding to the second image area in which the color within the first image area has been changed in the change process is generated as the teaching data.

3. The method for generating teaching data according to claim 1 or 2, characterized in that: In the generation process, The rapid deceleration or sudden stop of the moving body is detected, and the teaching data is generated based on the captured image of the moving body corresponding to the time of the rapid deceleration or sudden stop.

4. A teaching data generation device, characterized in that, Comprising: A determination unit for determining a second image area including a first image area for obtaining identification information of a moving body from a captured image; and A generation unit for generating data corresponding to the second image area determined by the determination unit as teaching data for machine learning, The captured image is a plurality of images continuously captured in a time series, The determination unit determines the size of the second image area based on the size of the first image area, and detects the movement of the first image area in the plurality of images, and determines the second image area corresponding to the movement of the first image area.

5. The teaching data generation device according to claim 4, characterized in that: It further comprises a change unit for changing the color within the first image area, The generation unit generates data corresponding to the second image area in which the color within the first image area has been changed as the teaching data.

6. The teaching data generation device according to claim 4 or 5, characterized in that: The generation unit detects the rapid deceleration or sudden stop of the moving body, and generates the teaching data based on the captured image of the moving body corresponding to the time of the rapid deceleration or sudden stop.

Images