Processing device, processing method, and program

Abstract

To achieve both of increase in accuracy of calculating information concerning the shape of an object being conveyed and reduction of a time delay in calculating the information concerning the shape of the object being conveyed.SOLUTION: When it is determined that there is a steel material 100 being conveyed in an imaging area of an imaging device 140, object shape information which is information concerning the shape of the steel material 100 is calculated using a trained model trained a relation between an image obtained by imaging the imaging area of the imaging device 140 and the object shape information based on an image for object imaging imaged at timing after timing of the determination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] There is a demand for technology that can automatically calculate information about the shape of an object being transported, such as a material being rolled on a rolling line. As an example of this type of technology, Patent Document 1 discloses a method for extracting a contour line of one end of a rolled material in the longitudinal direction from an image of the side of the rolled material, the contour line including the one end of the rolled material, and measuring a parameter representing the warpage shape of the one end from the extracted contour line.

[0003] Furthermore, Patent Document 2 discloses a method for determining the shape of a steel plate using a shape determination model generated by machine learning, in which an image of a steel plate rolled by a rolling mill is used as input data and shape determination information for determining the shape of the steel plate is used as output data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 1-320419 [Patent Document 2] JP 2022-181167 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, the contour line of an object is extracted by image processing such as binarization. Therefore, for example, depending on the shooting environment, there is a risk that the detection accuracy of the shape of the object may decrease. For example, in a rolling line, a large amount of disturbance such as water vapor may occur. If an image of a rolled material is captured at such a timing, it is not easy to extract the contour line of the rolled material with high accuracy.

[0006] In addition, the technology described in Patent Document 2 is premised on the assumption that an image is captured at the timing when an object is transported to the photographing area of ​​the photographing device. Therefore, for example, there is a risk of missing the timing of photographing an object being transported. For example, in a rolling line, a plurality of rolled materials are generally transported in sequence. In this case, it is not easy to accurately identify the timing when the rolled material is transported to the photographing area of ​​the photographing device. Therefore, it is necessary to always photograph the rolled material and calculate the shape discrimination information using the machine learning model. In addition, depending on the algorithm of the machine learning model, there is a risk that the time required to detect the shape of an object will be long. Therefore, there is a risk that the following rolled material will pass through the photographing area before the process for detecting the shape of the rolled material based on the image captured to detect the shape of the preceding rolled material is completed.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to achieve both high accuracy in calculating information about the shape of an object being transported and suppression of time delays in calculating information about the shape of an object being transported. [Means for solving the problem]

[0008] The processing device of the present invention comprises an acquisition unit that acquires an image captured in a shooting area that includes an object transport path and in which at least a portion of the side of the object can be captured, an object presence / absence determination unit that determines whether the object is present being transported through the shooting area, and a calculation unit that calculates object shape information, which is information regarding the shape of the object, based on the image captured at a time after the object presence / absence determination unit determines that the object is present in the shooting area, and the calculation unit calculates the object shape information using a trained model that has learned the relationship between the image captured of the shooting area and the object shape information.

[0009] The processing method of the present invention comprises an acquisition step of acquiring an image captured in a shooting area which is an area including a transport path for an object and in which at least a portion of the side of the object can be captured; an object presence / absence determination step of determining whether the object is present being transported through the shooting area; and a calculation step of calculating object shape information, which is information regarding the shape of the object, based on the image captured at a timing after the timing at which it is determined by the object presence / absence determination step that the object is present in the shooting area, and the calculation step calculates the object shape information using a trained model which has learned the relationship between the image captured in the shooting area and the object shape information.

[0010] The program of the present invention causes a computer to function as each part of the processing device. Effect of the Invention

[0011] According to the present invention, it is possible to improve both the accuracy and speed of calculation of information relating to the shape of an object being transported. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram showing an example of a steel material being transported to a position on the exit side of a rolling mill. [Diagram 2] FIG. 2 illustrates an example of a functional configuration of a processing device. [Diagram 3] FIG. 2 is a diagram showing an example of an image for photographing an object. [Figure 4] FIG. 2 is a diagram illustrating an example of an upstream area and a downstream area. [Diagram 5] 11 is a diagram illustrating an example of a relationship between a processing time per image of an image processing unit and a processing time per image of a calculation unit. FIG. [Figure 6] FIG. 11 is a diagram illustrating an example of a method for calculating the amount of upward warping of a steel material. [Figure 7] 11 is a diagram illustrating an example of a method for calculating a trajectory of a tip of a steel material. FIG. [Figure 8] FIG. 13 is a diagram showing an example of a trajectory of a tip of a steel material. [Figure 9]1 is a diagram illustrating how the trajectory of the tip of a steel material changes depending on the warp of the steel material. FIG. [Figure 10A] 11 is a flowchart illustrating an example of a processing method. [Figure 10B] 13 is a flowchart illustrating an example of step S1002. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In addition, the fact that the objects to be compared are the same in terms of length, position, size, interval, etc. includes not only the case where they are exactly the same, but also the case where they are different within the scope of the invention (for example, the difference within the tolerance range determined at the time of design). In addition, the XYZ coordinates shown in each figure indicate the relationship of the directions in each figure. The position of the origin of the XYZ coordinates is not limited to the position shown in each figure. In addition, as a symbol representing one axis of the XYZ coordinates, a symbol with a black circle (●) in a white circle (○) and a symbol with a cross mark (×) in a white circle (○) are used. A symbol with a black circle (●) in a white circle (○) represents an arrow line (axis) from the back side of the paper to the front side. A symbol with a cross mark (×) in a white circle (○) represents an arrow line (axis) from the front side of the paper to the back side. In addition, for convenience of explanation and notation, the configuration in each figure is omitted, simplified, and deformed as necessary.

[0014] [Object being transported] In this embodiment, the details of which will be described later, an image of an object being transported on a transport path is captured, and object shape information, which is information about the shape of the object being transported, is calculated based on the captured image. The object being transported is not limited, but in this embodiment, a case in which a steel material (e.g., a slab) being transported to a position on the exit side of a roughing mill in a hot rolling line is taken as an example.

[0015] Fig. 1 is a diagram showing an example of a steel material 100 being transported at the exit position of a rolling mill 110. Hereinafter, the rolling mill 110 in Fig. 1 will be described as a reverse rolling mill, which is one of the rolling mills constituting a roughing mill. The exit side of a rolling mill refers to the downstream side of the rolling mill in the transport direction of the steel material 100. The entry side of a rolling mill refers to the upstream side of the rolling mill in the transport direction of the steel material 100.

[0016] In each drawing, a case where the steel material 100 is transported in the positive direction of the X-axis is illustrated. Therefore, the position of the exit side of the rolling mill 110 is a position on the positive side of the X-axis from the rolling mill 110. Also, the position of the entry side of the rolling mill 110 is a position on the negative side (origin side) of the X-axis from the rolling mill 110. In the following description, the downstream side in the transport direction of the steel material 100 is abbreviated to the downstream side as necessary. Similarly, the upstream side in the transport direction of the steel material 100 is abbreviated to the upstream side as necessary (note that in each drawing, the negative side (origin side) of the X-axis is the upstream side). Also, in the following description, the downstream end 100a of the steel material 100 is referred to as the front end 100a of the steel material 100 as necessary, and the upstream end 100b of the steel material 100 is referred to as the rear end 100b of the steel material 100 as necessary.

[0017] 1 illustrates a case where the steel material 100 is transported by transport rolls provided on the transport tables 120a to 120f. Therefore, in FIG. 1, a case where the transport path 130 is a space above the transport tables 120a to 120f through which the steel material 100 is expected to pass is illustrated.

[0018] The rolling mill 110 in FIG. 1 is a reverse rolling mill that performs rolling on the steel material 100 multiple times by moving the steel material 100 back and forth in the conveying direction (X-axis direction). Each rolling is called a pass. When the rolling mill 110 performs three rolling (passes) on the steel material 100, for example, the first pass is performed by passing the steel material 100 between the upper and lower work rolls 110a and 110b of the rolling mill 110 from the upstream side of the rolling mill 110. When the first pass is completed, the rear end 100b of the steel material 100 may or may not reach a position on the exit side of the rolling mill 110. Thereafter, the second pass is performed by passing the steel material 100 between the upper and lower work rolls 110a and 110b of the rolling mill 110 from the downstream side of the rolling mill 110. At the end of the second pass, the front end 100a of the steel material 100 may or may not reach a position on the entry side of the rolling mill 110. Thereafter, a third pass is performed by passing the steel material 100 between the upper and lower work rolls 110a, 110b of the rolling mill 110 from the upstream side of the rolling mill 110. At the end of the third pass, the rear end 100b of the steel material 100 reaches a position on the exit side of the rolling mill 110. When the third pass is completed, the steel material 100 is transported downstream.

[0019] In each pass, the upper and lower work rolls 110a, 110b rotate while coming into contact with the steel material 100. The rotation direction of the upper and lower work rolls 110a, 110b is opposite to the traveling direction of the steel material 100 in the region where they come into contact with the steel material 100. For example, in odd-numbered passes, the work rolls 110a, 110b rotate counterclockwise and clockwise, respectively, as viewed in the plane of FIG. 1, and in even-numbered passes, the work rolls 110a, 110b rotate clockwise and counterclockwise, respectively, as viewed in the plane of FIG. 1.

[0020] The reverse rolling mill and the method of reverse rolling themselves are realized by known techniques and are not limited to those described above. For example, the number of passes in reverse rolling is not limited to three. In addition, another rolling mill constituting a roughing mill may be installed at least on one of the upstream side and downstream side of the rolling mill 110. The number of the other rolling mills may be one or more. In addition, the other rolling mill may not be a reverse rolling mill, and may be a reverse rolling mill. In addition, in the embodiment of FIG. 1, the rolling mill 110 is a reverse rolling mill, but instead of a reverse rolling mill, a normal rolling mill (a rolling mill that performs rolling while transporting the steel material 100 only in the direction toward the downstream side) may be used.

[0021] The hot rolling line itself can also be realized by a known technique. For example, a heating furnace (not shown) for heating the steel material 100 transported from the yard and a sizing press (not shown) for correcting the widthwise shape and length of the steel material 100 after heating in the heating furnace may be installed upstream of the rough rolling mill. Also, a finishing rolling mill for finish rolling the steel material 100 after rough rolling, a cooling device for cooling the steel material 100 after finish rolling on a run-out table, and a winding device for winding the steel material 100 after cooling in the cooling device may be installed downstream of the rough rolling mill.

[0022] As described above, in this embodiment, an image of an object being transported along the transport path is captured. At this time, an area including the transport path where at least a part of the side surface of the object can be captured is set as the capture area. This is because the shape of the object can be determined from the shape of the side surface of the object.

[0023] The entire region of the side surface of the object may be included in the photographing region. However, for example, when the object is long, it may not be easy to include the entire region of the side surface of the object in the photographing region. Also, it may be sufficient to determine the shape of a part of the region of the side surface of the object. For example, it may be sufficient to determine the shape of a region of the end of the object that includes one end of the object in the conveying direction. Therefore, the photographing region may be an area in which a part of the side surface of the object can be photographed.

[0024] In the example shown in FIG. 1, the side surfaces 100c, 100d of the steel material 100 are the ends of the steel material 100 in the width direction (Y-axis direction). In this embodiment, a case is exemplified in which an image of a partial area of ​​one side surface 100c of the steel material 100 being transported on the transport path 130 is captured using the imaging device 140. Although details will be described later, this embodiment exemplifies a case in which object shape information, which is information on the shape of the object being transported, includes the amount of warping of the tip portion 100a of the steel material 100. Therefore, this embodiment exemplifies a case in which the imaging area of ​​the imaging device 140 is a partial area of ​​the side surface 100c of the steel material 100, and is an area in which an area including the tip portion 100a of the steel material 100 can be captured. Note that the steel material 100 captured by the imaging device 140 is glowing due to red heat.

[0025] In this embodiment, the case is illustrated in which the photographing device 140 is installed so that the steel material 100 does not enter the photographing area of ​​the photographing device 140 before the final pass (the third pass in this embodiment) in the reverse rolling is completed, and the steel material 100 transported to the downstream side after the final pass is completed is included in the photographing area. In this way, for example, the steel material 100 shown in the image photographed by the photographing device 140 can be only the steel material 100 after the reverse rolling is completed (after all passes are completed). Therefore, it is not necessary to determine which pass the steel material 100 shown in the image photographed by the photographing device 140 is the steel material 100 after it has been completed. However, this is not necessarily required. For example, the photographing device 140 may be installed so that the steel material 100 after each pass is completed is included in the photographing area each time. Also, for example, a plurality of photographing devices 140 may be installed as the photographing device 140 for photographing the steel material 100 after each pass is completed.

[0026] In this embodiment, the photographing device 140 is installed so that the side surface 100c of the steel material 100 is photographed from substantially the front (preferably the front) (i.e., the photographing device 140 is substantially directly facing (preferably directly facing) the side surface 100c of the steel material 100). In addition, in this embodiment, the photographing device 140 is installed so that the horizontal direction of the image photographed by the photographing device 140 and the conveying direction of the steel material 100 are substantially parallel (preferably parallel). By doing so, for example, it is possible to omit processing such as changing the orientation of the image photographed by the photographing device 140. Also, it becomes easier for the inspector to check the state of the steel material 100 shown in the image photographed by the photographing device 140. However, this is not necessarily required. In this case, for example, by performing projective transformation on the image captured by the photographing device 140, at least one of the following may be performed: the image captured by the photographing device 140 is an image of the side 100c of the steel material 100 captured from approximately the front (preferably the front), and the horizontal direction of the image captured by the photographing device 140 is approximately parallel (preferably parallel) to the transport direction of the steel material 100.

[0027] Furthermore, in this embodiment, a case is exemplified in which no enclosure is installed to enclose the photographing area of ​​the photographing device 140 as long as the transportation of the steel material 100 is not impeded. In this way, for example, the cost of the equipment can be reduced. However, such an enclosure may be installed. As will be described later, in this embodiment, even without such an enclosure, it is possible to reduce the influence of disturbances such as water vapor on the calculation results of information relating to the shape of the steel material 100 (object shape information). However, if such an enclosure is installed, the influence can be further reduced.

[0028] Moreover, in this embodiment, a case where the imaging device 140 captures moving images at a predetermined frame rate (fps: frames per second) is exemplified. However, this is not necessarily required. For example, the imaging device 140 may capture still images at a predetermined time interval. Furthermore, in this embodiment, a case where the imaging device 140 captures color images is exemplified. However, this is not necessarily required. For example, the imaging device 140 may capture grayscale images. In the present embodiment, the image capturing device 140 is a network camera (IP camera), but the image capturing device 140 may be a camera other than a network camera.

[0029] 1(b), the present embodiment illustrates a case where a sensor 150 is installed to detect the passage of a leading end 100a of a steel material 100 being transported through an area between the rolling mill 110 and the imaging device 140. The sensor 150 may be, for example, an HMD (Hot Metal Detector) or a sensor equipped with a light-emitting unit and a light-receiving unit. The present embodiment illustrates a case where the sensor 150 is installed at a position where it can detect the passage of the leading end 100a of the steel material 100 after the first pass has been completed.

[0030] As described above, the object being transported is not limited to a steel material (e.g., a slab) being rough rolled by a rough rolling mill in a hot rolling line. For example, the object being transported may be a steel material being rolled by a rolling mill in a cold rolling line. The object being transported may also be a steel slab being transported in a continuous casting line (a steel material cut from a slab continuously cast by a continuous casting machine). The object being transported is not limited to a steel material.

[0031] [Processing device] 2 is a diagram showing an example of a functional configuration of the processing device 200. The processing device 200 has, as hardware, one or more hardware processors such as a CPU (Central Processing Unit) and one or more memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and executes various calculations by executing one or more programs stored in the memory by the one or more hardware processors. Furthermore, the processing device 200 has, as hardware, an input device and an output device.

[0032] 2, in this embodiment, the processing device 200 includes an acquisition unit 210, an image processing unit 220, an object presence / absence determination unit 230, a calculation start condition determination unit 240, a storage unit 250, a calculation unit 260, and an output unit 270. An example of the functions of each unit will be described below.

[0033] <Acquisition part 210> The acquisition unit 210 acquires various pieces of information necessary for the processing device 200 to calculate object shape information of the steel material 100. In this embodiment, an example is shown in which the information acquired by the acquisition unit 210 includes an image captured by the imaging device 140 and object identification information. In the following description, an image captured by the imaging device 140 to obtain an image of the steel material 100 for which object shape information is to be calculated is referred to as an object imaging image as necessary.

[0034] As described above, in this embodiment, the photographing device 140 is a network camera, and the photographing device 140 photographs color moving images at a predetermined frame rate. Therefore, in this embodiment, the photographing device 140 transmits color moving images (frame images) photographed at a predetermined frame rate to the processing device 200, and the acquiring unit 210 acquires images for photographing an object by receiving each frame image transmitted from the photographing device 140.

[0035] FIG. 3 is a diagram showing an example of an object photographing image (one frame image). FIG. 3(a) is a diagram showing an example of an object photographing image 300a showing a steel material 100 being transported in the photographing area of ​​the photographing device 140. FIG. 3(b) is a diagram showing an example of an object photographing image 300b when the steel material 100 is not present in the photographing area of ​​the photographing device 140. Thus, in this embodiment, a case is illustrated in which the photographing device 140 photographs the object photographing images 300a and 300b regardless of whether the steel material 100 is present in the photographing area of ​​the photographing device 140. Therefore, in this embodiment, a case is illustrated in which the object photographing images acquired by the acquisition unit 210 include an object photographing image 300a including the steel material 100 and an object photographing image 300b including no steel material 100.

[0036] In addition, as described later, in this embodiment, a case where image processing using a background difference method is performed is exemplified. Therefore, in this embodiment, a case where the acquisition unit 210 acquires a background image other than the object shooting images 300a and 300b in advance (before acquiring the object shooting image) as an image to be shot by the shooting device 140 when the steel material 100 does not exist in the shooting area of ​​the shooting device 140 is exemplified. Note that the background image is an image of the background (shooting area of ​​the shooting device 140) that does not include the object to be detected (steel material 100 in this embodiment). In addition, the background image may be updated, for example, periodically, or may be updated when at least one of the operating conditions, the environment around the object (steel material 100 in this embodiment) (for example, at least one of weather and the presence or absence of lighting), and the attributes of the steel material 100 (for example, at least one of type, width, length, and thickness) has changed. Note that the environment around the object (steel material 100 in this embodiment) is, for example, a factor that affects the object shown in the shot image, and includes information on the atmospheric state around the object.

[0037] The object identification information is identification information of an object for which object shape information is to be calculated. In this embodiment, an example is shown in which the object identification information is identification information of a steel material 100 being rolled by a rolling mill 110. In the following description, the identification information of the steel material 100 is referred to as a steel material number as necessary.

[0038] In this embodiment, a case is exemplified where which steel material 100 is located at which position in the hot rolling line is managed by an operation management device (external device) (not shown) that manages the operation in the hot rolling line. The position of the steel material 100 in the hot rolling line is specified, for example, by so-called tracking. Tracking is performed, for example, by calculating the distance from the most upstream position of the hot rolling line to the position of a predetermined part of the steel material 100 (for example, the tip portion 100a) at a predetermined time period based on the conveying speed of the steel material 100 and the elapsed time from the start of conveying the steel material 100 in the hot rolling line. For example, the operation management device manages which steel material 100 is located at which position in the hot rolling line by storing the steel material number and the position of the steel material 100 of the steel material number in association with each other. Note that the management of the position of the steel material 100 by tracking itself can be realized by a known technology, and the tracking method is not limited to the above-mentioned method.

[0039] In this embodiment, an example is shown in which the operation management device identifies, by tracking, the steel material number of the steel material 100 that is to undergo a first pass in the rolling mill 110 (reverse rolling mill), and then, when the tip portion 100a of the steel material 100 is detected for the first time by the sensor 150, transmits the steel material number to the processing device 200. Also, in this embodiment, an example is shown in which the acquisition unit 210 receives the steel material number transmitted from the operation management device in this manner, thereby acquiring the steel material number (object identification information).

[0040] The information acquired by the acquisition unit 210 is not limited to the object photographing image and object identification information (steel photographing image and steel number). For example, the acquisition unit 210 may acquire tracking information (information indicating the above-mentioned tracking content) in addition to the object identification information (steel number), or may acquire a detection signal in the sensor 150. Furthermore, the operation management device may determine that the tip portion 100a of the steel material 100 has passed a position between the rolling mill 110 and the photography device 140 for the first time based on the tracking result without using the detection result of the sensor 150. Furthermore, the processing device 200 may have at least a part of the functions of the above-mentioned operation management device.

[0041] <Image processing unit 220> The image processing unit 220 performs image processing on the object photographing image. In this embodiment, the image processing unit 220 can perform image processing on one object photographing image at the same speed as the frame rate of the object photographing image or at a speed equal to or higher than the frame rate of the object photographing image. For example, when the frame rate of the object photographing image is 60 fps, the image processing unit 220 performs image processing on one object photographing image in 1 / 60 sec or in a time shorter than 1 / 60 sec. However, this is not necessarily required. That is, the image processing unit 220 may perform image processing on one object photographing image at a speed slower than the frame rate of the object photographing image. However, in this case, it is preferable to end image processing on all object photographing images photographed by the photographing device 140 when the steel material 100 to be calculated for the object shape information is being reverse-rolled before the rolling mill 110 starts reverse rolling the steel material 100 next to the steel material 100 to be calculated for the object shape information.

[0042] As described above, in this embodiment, the case where the object photographing images acquired by the acquisition unit 210 include an object photographing image including the steel material 100 and an object photographing image not including the steel material 100 is exemplified. Therefore, in this embodiment, the case where the image processing unit 220 performs image processing on each object photographing image regardless of whether the object photographing image includes the steel material 100 or not is exemplified. Then, in this embodiment, the case where the image processing unit 220 performs image processing for improving the determination accuracy when the object presence / absence determination unit 230 described later determines whether the object photographing image includes the steel material 100 or not is exemplified. Therefore, in this embodiment, the case where the image processing unit 220 performs image processing for extracting at least one of the area of ​​the steel material 100 and the area other than the steel material 100 from the object photographing image is exemplified. A specific example of such image processing will be described below.

[0043] First, the image processing unit 220 converts the object photographing image and the background image into grayscale images. If the object photographing image and the background image are grayscale images, this process is not necessary. In the following description of a specific example of image processing in the image processing unit 220, it is assumed that the object photographing image and the background image are grayscale images.

[0044] Next, the image processing unit 220 creates a difference image between the object photographing image and the background image. For example, the image processing unit 220 calculates, for each pixel, a pixel value of the pixel in the object photographing image by subtracting the pixel value of the pixel in the background image corresponding to the pixel from the pixel value (tone value) of the pixel in the object photographing image, and sets the pixel value of the pixel in the difference image as the pixel value of the pixel. Since the difference image itself is realized by a known technology, a detailed description thereof will be omitted here.

[0045] Next, the image processing unit 220 smoothes the difference image. For example, a Gaussian filter can be used as a smoothing filter for smoothing, but the smoothing filter used for smoothing is not limited to a Gaussian filter. The smoothing process itself can be realized by a known technique, and therefore a detailed description thereof will be omitted here.

[0046] Next, the image processing unit 220 creates a binary image by binarizing each pixel value of the smoothed difference image. For example, the image processing unit 220 creates a binary image by performing the following for each pixel: if the pixel value exceeds a binarization threshold, the pixel value is set to "1"; otherwise, the pixel value is set to "0." The binarization process itself is realized by a known technique, and therefore a detailed description thereof will be omitted here.

[0047] Next, the image processing unit 220 performs a noise reduction process to reduce noise contained in the binarized image. An example of the noise reduction process is morphological transformation. The image processing unit 220 may perform at least one of dilation, erosion, opening, and closing as the morphological transformation. The image processing unit 220 may also perform top hat processing or black hat processing. The morphological transformation itself is realized by a known technique, and therefore a detailed description thereof will be omitted here. In the following description of a specific example of image processing in the image processing unit 220, the binarized image is assumed to be a binarized image after noise reduction processing.

[0048] Next, the image processing unit 220 extracts a plurality of regions one by one by dividing the image (binarized image in this embodiment) captured by the imaging device 140. The positions and number of the plurality of regions are determined in advance, for example, based on the surrounding environment of the object (steel material 100 in this embodiment) and the transport direction of the object in the image. In this embodiment, a case is illustrated in which the transport direction (X-axis direction) of the steel material 100 and the horizontal direction of the image for photographing the object are approximately parallel. In addition, when rolling (reverse rolling) is completed and the steel material 100 is transported downstream (positive direction of the X-axis) of the imaging area of ​​the steel material 100, water vapor may be generated from the downstream side of the imaging area of ​​the steel material 100 toward the imaging area of ​​the steel material 100 due to removal of scale (oxide) on the surface of the steel material 100, etc. In this embodiment, in order to easily distinguish between the water vapor and the steel material 100, a case is illustrated in which a binary image 400 is divided into two areas, an upstream area 410 (an area on the negative side of the X-axis) and a downstream area 420 (an area on the positive side of the X-axis), as shown in Fig. 4. For convenience of notation, Fig. 4 illustrates an example in which the pixel value of the black area (the area of ​​the steel material 100) in the binary image 400 is "1" and the pixel value of the white area is "0".

[0049] For example, as illustrated in FIG. 4, if there is a large area in the upstream area 410 where the pixel value is "1" (a black area in FIG. 4) and there is no large area in the downstream area 420 where the pixel value is "1", it can be determined that the binarized image 400 contains steel material 100. This is because, when the binarized image 400 contains steel material 100, the size of the area with pixel value "1" changes over time in the following order: upstream area 410 > downstream area 420, upstream area 410 = downstream area 420, downstream area 420 > upstream area 410. On the other hand, if there is a large area in the downstream area 420 where the pixel value is "1" and there is no large area in the upstream area 410 where the pixel value is "1", it can be determined that the binarized image 400 does not contain steel material 100. In this case, it can be determined that water vapor is being generated. This is because when water vapor is generated, the size of the area with a pixel value of "1" changes over time in the following order: downstream area 420 > upstream area 410, upstream area 410 = downstream area 420, upstream area 410 > downstream area 420.

[0050] Image processing unit 220 calculates the blob area for each of a plurality of regions obtained by dividing an image (binarized image) captured by imaging device 140. The blob area is the sum of the areas of the blobs. A blob is a group of pixels in the binary image having a pixel value corresponding to an object to be detected (for example, pixels having a pixel value of "1" in the binary image (pixels in a black region in FIG. 4)). In the binarization process, blob analysis (calculation of the feature amount of each blob, labeling of each blob, etc.) is generally performed. However, in this embodiment, for example, the number of pixels having a pixel value of "1" in binary image 400 can be calculated as the blob area, so blob analysis does not need to be performed.

[0051] In the present embodiment, a case will be illustrated in which the image processor 220 calculates the blob area in the upstream area 410 and the blob area in the downstream area 420 as such blob areas. For example, when the environment around the object (steel material 100 in this embodiment) is good (an environment in which the object is clearly reflected in the captured image), it is not necessary to divide the image (binarized image) captured by the imaging device 140 into multiple regions. In this case, the image processing unit 220 may calculate, for example, the entire blob area of ​​the image (binarized image) captured by the imaging device 140.

[0052] <Object presence / absence determination unit 230> The object presence / absence determination unit 230 determines whether or not there is an object being transported through the photographing area of ​​the photographing device 140. In this embodiment, a case is exemplified in which the object presence / absence determination unit 230 determines whether or not there is an object being transported through the photographing area of ​​the photographing device 140 based on whether or not an object is included in the image photographed by the photographing device 140. Although it may be possible to determine whether or not there is an object being transported through the photographing area of ​​the photographing device 140 using the image photographed by the photographing device 140 itself, there is a risk of the accuracy of the determination being reduced. Therefore, in this embodiment, a case is exemplified in which the object presence / absence determination unit 230 determines whether or not an object is included in the image photographed by the photographing device 140 based on the result of image processing by the image processing unit 220. In addition, in order to further increase the accuracy of the determination, in this embodiment, a case is exemplified in which the object presence / absence determination unit 230 determines whether or not an object is included in the image photographed by the photographing device 140 based on the result of image processing for each of a plurality of areas into which the image photographed by the photographing device 140 (binary image) is divided. A specific example of the process of the object presence / absence determination unit 230 in such a case will be described below.

[0053] The object presence / absence determination unit 230 determines whether or not the blob area is within a predetermined range for each of a plurality of regions obtained by dividing the object photographing image (binarized image). The predetermined range is determined in advance for each of the plurality of regions based on, for example, the surrounding environment of the object (steel material 100 in this embodiment) and the size of the object.

[0054] In this embodiment, an example is shown in which the object presence / absence determination unit 230 determines whether the area of ​​a blob in the upstream area 410 is within a predetermined range defined for the upstream area 410, and determines whether the area of ​​a blob in the downstream area 420 is within a predetermined range defined for the downstream area 420.

[0055] As described above, in this embodiment, if there is a large area in the upstream area 410 where the pixel value is "1" (a black area in FIG. 4) and there is no large area in the downstream area 420 where the pixel value is "1", it can be determined that the binarized image 400 contains steel material 100. On the other hand, if there is a large area in the downstream area 420 where the pixel value is "1" and there is no large area in the upstream area 410 where the pixel value is "1", it can be determined that the binarized image 400 does not contain steel material 100.

[0056] Therefore, in this embodiment, a case is illustrated in which the predetermined range in the upstream area 410 and the predetermined range in the downstream area 420 are respectively determined so that the lower limit value of the predetermined range in the upstream area 410 is greater than the lower limit value of the predetermined range in the downstream area 420. Also, a case is illustrated in which the upper limit values ​​of the predetermined ranges in the upstream area 410 and the downstream area 420 are determined so as to suppress determination that the steel material 100 is contained in the binarized image 400 when a large amount of water vapor is contained and the area with pixel values ​​of "1" is abnormally large.

[0057] Next, the object presence / absence determining section 230 determines whether or not the blob area is within a predetermined range in all of the multiple regions obtained by dividing the object photographing image (binarized image). In this embodiment, an example is shown of a case in which the object presence / absence determination unit 230 determines whether both of the following are true: the area of ​​a blob in the upstream area 410 is within a predetermined range defined for the upstream area 410, and the area of ​​a blob in the downstream area 420 is within a predetermined range defined for the downstream area 420.

[0058] If the result of this determination is that the blob area is within a predetermined range in all of the multiple regions into which the image for photographing an object (binary image) is divided, the object presence / absence determination unit 230 determines that the image for photographing an object (binary image) contains an object. On the other hand, if the blob area is not within the predetermined range in at least one of the multiple regions into which the image for photographing an object (binary image) is divided, the object presence / absence determination unit 230 determines that the image for photographing an object (binary image) does not contain an object.

[0059] In this embodiment, a case is illustrated in which the object presence / absence determination unit 230 determines that the steel material 100 is included in the image for photographing an object (binarized image) when the area of ​​the blob in the upstream area 410 is within a predetermined range defined for the upstream area 410 and the area of ​​the blob in the downstream area 420 is within a predetermined range defined for the downstream area 420. On the other hand, a case is illustrated in which the object presence / absence determination unit 230 determines that the steel material 100 is not included in the image for photographing an object (binarized image) when at least one of the following is true: the area of ​​the blob in the upstream area 410 is not within a predetermined range defined for the upstream area 410 and the area of ​​the blob in the downstream area 420 is not within a predetermined range defined for the downstream area 420.

[0060] As described above, in this embodiment, the case where the object presence / absence determination unit 230 determines whether or not there is a steel material 100 being transported through the imaging area of ​​the imaging device 140 based on the result of image processing on the image for photographing the object is exemplified. In this way, it is possible to determine whether or not there is a steel material 100 being transported through the imaging area of ​​the imaging device 140 in a short period (for example, the period of the frame rate of the imaging device 140) and with high accuracy. However, this is not necessarily required. For example, the object presence / absence determination unit 230 may determine whether or not there is a steel material 100 being transported through the imaging area of ​​the imaging device 140 based on the result of the tracking described above. Also, a sensor may be installed that detects that the tip portion 100a of the steel material 100 being transported has passed a position upstream of the imaging area of ​​the imaging device 140. In this case, the object presence / absence determination unit 230 may determine whether or not there is a steel material 100 being transported through the imaging area of ​​the imaging device 140 based on the detection signal of the sensor. As described above, when determining whether or not there is steel material 100 being transported through the shooting area of ​​the shooting device 140, if the results of image processing on the object shooting image are not used, the processing device 200 does not need to be equipped with the image processing unit 220.

[0061] <Calculation start condition determination unit 240> Calculation start condition determination section 240 determines whether or not a calculation start condition for object shape information by calculation section 260 (a condition for calculation section 260 to start calculating object shape information, which will be described later) is satisfied.

[0062] The calculation start condition includes, for example, a condition that the rear end of an object transported immediately before the object (steel material 100 in this embodiment) for which the object shape information is to be calculated can be considered to be transported downstream of the photographing area of ​​the photographing device 140. In this embodiment, the calculation start condition is exemplified as a certain time having elapsed since the object (steel material 100 in this embodiment) for which the object shape information is to be calculated is detected at a predetermined position upstream of the photographing area of ​​the photographing device 140. As described above, in this embodiment, the operation management device specifies the steel material number of the steel material 100 for which the first pass is to be performed, and then transmits the steel material number to the processing device 200 when the tip portion 100a of the steel material 100 is detected for the first time by the sensor 150. Therefore, in this embodiment, the calculation start condition is exemplified as a certain time having elapsed since the object identification information (steel material number) is (newly) acquired by the acquisition unit 210 and the object identification information (steel material number) was acquired.

[0063] The certain time is determined in advance based on, for example, the time assumed to be required for the rolling of the steel material 100 by the rolling mill 110. In the case of reverse rolling, the time may be, for example, the time from when the front end 100a of the steel material 100 is first detected by the sensor 150 in the first pass until the rear end 100b of the steel material 100 is no longer detected by the sensor 150 in the last pass. For example, based on at least one of operational records, simulation experiments, and computer simulations, the shortest time assumed to be required for the rolling of the steel material 100 (reverse rolling) by the rolling mill 110 may be derived, and the shortest time or a time shorter than the shortest time by a margin may be determined as the certain time. In addition, the certain time may be determined for at least one of the operational conditions (e.g., conveying speed) and the attributes of the steel material 100 (e.g., at least one of the type, width, length, and thickness), or may be the same regardless of these.

[0064] In this manner, by the calculation start condition determination unit 240 determining whether or not the calculation start condition is satisfied, it is possible to prevent the calculation unit 260 from calculating object shape information of the object identified by the object identification information (steel product number) acquired by the acquisition unit 210 based on an object photography image of the object (steel product 100 in this embodiment) that was transported immediately before the object identified by the object identification information (steel product number) acquired by the acquisition unit 210.

[0065] The calculation start condition is not limited to a certain time having passed since the object (steel material 100 in this embodiment) for which object shape information is to be calculated is detected at a predetermined position upstream of the imaging area of ​​the imaging device 140. For example, a sensor may be installed that detects the passage of the rear end 100b of the steel material 100 being transported at a position downstream of the imaging area of ​​the imaging device 140. In this case, the calculation start condition may be the detection of the passage of the rear end 100b of the steel material 100 by the sensor (for example, the steel material 100 goes from being detected by the sensor to not being detected by the sensor).

[0066] Also, for example, when the interval between two consecutive objects in the transport order is long, and it is certain that the rear end of the object transported immediately before the object identified by the object identification information (steel material No. in this embodiment, steel material 100) is transported downstream of the photographing area of ​​the photographing device 140 at the timing when the acquiring unit 210 acquires the object identification information (steel material No.), for example, the calculation start condition may be only that the object identification information (steel material No.) is (newly) acquired by the acquiring unit 210. Also, for example, when it is not necessary to associate the object identification information with the object shape information, it is not necessary to judge whether the calculation start condition is satisfied. In this case, the processing device 200 may not be provided with the calculation start condition judgment unit 240.

[0067] <Storage unit 250> The storage unit 250 stores (records) images captured (acquired by the acquisition unit 210) after the timing when the object presence / absence determination unit 230 determines that there is an object being transported through the shooting area of ​​the imaging device 140. In this embodiment, a case is illustrated in which the storage unit 250 stores images for photographing an object captured (acquired by the acquisition unit 210) after the timing when the object presence / absence determination unit 230 determines that there is an object being transported through the shooting area of ​​the imaging device 140. In addition, in this embodiment, a case is illustrated in which the storage unit 250 starts storing images for photographing an object acquired by the acquisition unit 210 when the calculation start condition determination unit 240 determines that the calculation start condition is satisfied. In this case, if the calculation start condition determination unit 240 does not determine that the calculation start condition is satisfied, the storage unit 250 does not start storing images for photographing an object acquired by the acquisition unit 210.

[0068] The images (images for photographing an object) stored by the storage unit 250 are used for the calculation of object shape information by the calculation unit 260, which will be described later. Therefore, the storage unit 250 only needs to store images (images for photographing an object) capturing a portion of the object for which object shape information is to be calculated (in this embodiment, a region including the tip 100a of the steel material 100). Therefore, in this embodiment, a case is illustrated in which the storage unit 250 stores images for photographing an object for a predetermined time (a predetermined number of frames). The predetermined time is determined in advance, for example, based on the length of the portion of the object for which object shape information is to be calculated, the transport speed of the object, and the frame rate. In the following description, this predetermined time will be referred to as the storage time, as necessary.

[0069] <Calculation unit 260> When the object presence / absence determination unit 230 determines that an object is present in the shooting area of ​​the image capture device 140, the calculation unit 260 calculates object shape information, which is information about the shape of the object, based on the image acquired by the acquisition unit 210. The calculation unit 260 uses a trained model that has learned the relationship between the image captured in the shooting area of ​​the image capture device 140 and the object shape information, in calculating the object shape information. In this embodiment, a case where the trained model is YOLO (You Only Look Once) v5 is exemplified. However, the trained model is not limited to YOLO (more specifically, YOLO v5). For example, the trained model may be other object detection algorithms such as R-CNN (Region convolutional neural network) and SSD (Single Shot MultiBox Detector). In addition, the trained model may be other artificial intelligence, deep learning models, or machine learning models. The trained model may be a classifier. Note that the learning method of the trained model itself can be realized by a known technology, so a detailed description thereof will be omitted here.

[0070] In addition, in this embodiment, a case is illustrated in which the calculation unit 260 calculates object shape information based on each of the object photographing images stored by the storage unit 250 after the timing when the calculation start condition determination unit 240 determines that the calculation start condition is satisfied. In addition, in this embodiment, a case is illustrated in which the time required for the calculation unit 260 to calculate object shape information based on one object photographing image is longer than the time required for the image processing unit 220 to perform image processing on one object photographing image (the time required from creating a grayscale image to calculating the blob area). In the following description, the time required for the calculation unit 260 to calculate object shape information based on one object photographing image is referred to as the processing time per image of the calculation unit 260 as necessary. In addition, the time required for the image processing unit 220 to perform image processing on one object photographing image is referred to as the processing time per image of the image processing unit 220 as necessary.

[0071] FIG. 5 is a diagram illustrating an example of the relationship between the processing time Δt1 per image of the image processing unit 220 and the processing time Δt2 per image of the calculation unit 260. In FIG.

[0072] By using a trained model such as YOLO v5, it is possible to reduce the influence of disturbances such as water vapor on the calculation results of information (object shape information) relating to the shape of the steel material 100 (compared to the case where the shape of the object is extracted by image processing). On the other hand, using a trained model such as YOLO v5 increases the calculation time (compared to the case where the shape of the object is extracted by image processing).

[0073] When the image processing unit 220 processes one object photographing image at the same speed as the frame rate of the object photographing image, the processing time Δt1 per image of the image processing unit 220 is, for example, the reciprocal of the frame rate of the object photographing image. In this case, in the example shown in Fig. 5, the processing time Δt2 per image of the calculation unit 260 is three times the reciprocal of the frame rate of the object photographing image.

[0074] Therefore, if the calculation unit 260 calculates the object shape information based on the object photographing image every time the acquisition unit 210 acquires the object photographing image, there is a risk that rolling of the next steel material 100 of the steel material 100 is started before the calculation of the object shape information based on all the object photographing images for the steel material 100 for which the object shape information is to be calculated is completed. Therefore, a time delay occurs in the calculation of the object shape information. In order to prevent such a time delay from occurring, it is considered that the calculation unit 260 calculates the object shape information based on only a part of the object photographing images acquired by the acquisition unit 210 (for example, the object photographing images acquired at a time interval of the processing time Δt2 per image by the calculation unit 260). For example, in the example shown in FIG. 5, it is considered that the calculation unit 260 calculates the object shape information based on the object photographing images acquired by the acquisition unit 210 at times t1, t4, and t7. However, if this is done, the time interval (period) for calculating the object shape information becomes longer. In this case, it is not possible to calculate object shape information that should be calculated (object shape information based on the object photographing images photographed (acquired by the acquisition unit 210) at times t2, t3, t5, and t6). Therefore, it is not possible to detect the object shape at the timing when it should be detected. Therefore, there is a risk that the detection accuracy of the object shape will decrease.

[0075] Therefore, in this embodiment, the calculation unit 260 calculates object shape information based on the object photographing image photographed (acquired by the acquisition unit 210) at a timing after the timing when the object presence / absence determination unit 230 determines that an object (steel material 100 in this embodiment) is present in the photographing area of ​​the photographing device 140. In this way, the object photographing image of the object for which the object shape information is to be calculated can be limited to object shape information including the object (steel material 100 in this embodiment). Therefore, before the object next to the object for which the object shape information is to be calculated (steel material 100 in this embodiment) is transported to the photographing area of ​​the photographing device 140, it is possible to increase the possibility that the object shape information based on all object photographing images including the object for which the object shape information is to be calculated can be calculated (preferably, the possibility can be made 0 (zero)). Therefore, it is possible to reduce the possibility that the shape of the object cannot be detected at the timing when it should be detected (preferably, the possibility can be made 0 (zero)). In addition, the calculation unit 260 calculates the object shape information based on an object photographing image captured (acquired by the acquisition unit 210) after the timing at which the calculation start condition determination unit 240 determines that the calculation start condition is satisfied, thereby further improving the calculation accuracy of the object shape information.

[0076] Note that improving the calculation accuracy of object shape information corresponds to a decrease in the discrepancy between information on the shape represented (quantitatively or qualitatively) based on the object shape information and the actual shape. 5, for convenience of notation, the processing time Δt2 per image of the calculation unit 260 is set to three times the processing time Δt1 per image of the image processing unit 220. However, the processing time Δt2 per image of the calculation unit 260 is not limited to three times the reciprocal of the frame rate of the object photographing image.

[0077] As described above, the object shape information is information related to the shape of an object. In this embodiment, the object shape information includes information that quantifies and expresses the shape of the object (steel material 100 in this embodiment) along the longitudinal direction, and information that indicates the change in the position of the object (steel material 100 in this embodiment) over time.

[0078] The information quantifying and expressing the shape of an object (steel material 100 in this embodiment) along the longitudinal direction includes, for example, parameters (numerical values) for identifying the shape. Specifically, the information quantifying and expressing the shape of an object (steel material 100 in this embodiment) along the longitudinal direction includes, for example, information quantifying and expressing the degree of deviation (for example, flatness) of the surface of the object. The information quantifying and expressing the degree of flatness of the surface of the object includes, for example, information quantifying and expressing the degree of warping such as upward warping and downward warping. The upward warping is a warping that gradually increases in position toward the tip of the object (tip 100a of the steel material 100 in this embodiment), with the tip of the object being at its highest. The downward warping is a warping that gradually decreases in position toward the tip of the object (tip 100a of the steel material 100 in this embodiment), with the tip of the object being at its lowest. In addition, there is also a warp (so-called sickle-shaped warp) in which the position is lowest or highest at the rear end side of the object from the front end (in this embodiment, the rear end side 100b from the front end 100a of the steel material 100) and the position becomes higher or lower toward the front end of the object. Furthermore, the information that quantifies and represents the shape of the object along the longitudinal direction is not limited to information that quantifies and represents the shape at the front end of the object. For example, the information that quantifies and represents the shape of the object along the longitudinal direction may be information that quantifies and represents the shape at the rear end of the object, or information that quantifies and represents the shape at the middle part of the object (part away from the front and rear ends in the longitudinal direction). For example, the information that quantifies and represents the shape of the object along the longitudinal direction may be information that indicates the degree of ear wave or center elongation.

[0079] In this embodiment, the case is illustrated where the information that quantifies and represents the shape along the longitudinal direction of the object is information that indicates the amount of upward warping (degree of upward warping) of the steel material 100. FIG. 6 is a diagram for explaining an example of a method for calculating the amount of upward warping of the steel material 100. In FIG. 6, a reference line 610 is set on an image 600 for photographing an object. The reference line 610 is, for example, a line that indicates a position that is assumed to be the position of the upper surface (upper surface) of the steel material 100 when it is assumed that the steel material 100 is flat. The reference line 610 may be set by an inspector inputting information indicating the reference line 610 to an input device of the processing device 200. The calculation unit 260 may also automatically set the reference line 610. For example, the calculation unit 260 may set the reference line 610 at a position in the image for photographing an object, which is a position that corresponds to the lowest position of the transport path 130, and is above the position by a length corresponding to the thickness of the steel material 100. In this case, the acquisition unit 210 may acquire information indicating an estimated value of the thickness of the steel material 100 after rolling, for example, from an operation management device. Then, the calculation unit 260 may convert the estimated value into a length in the object photographing image, and use the converted value as the length corresponding to the thickness of the steel material 100, thereby setting the reference line 610 as described above.

[0080] The calculation unit 260 calculates the height of the tip 100a of the steel material 100 with respect to the reference line 610 as the upward warping amount Δy of the steel material 100. In this embodiment, the case where the trained model is YOLO v5 is illustrated. In YOLO, a detected object is represented using a bounding box 620. The bounding box 620 is a rectangular (e.g., circumscribed rectangle) area surrounding the detected object. Therefore, the calculation unit 260, for example, specifies the pixel of the vertex 630 on the downstream side (positive direction side of the X-axis) and upper side (positive direction side of the Z-axis) of the conveying direction of the steel material 100 among the four vertices of the bounding box 620. In this case, the calculation unit 260 calculates the length (number of pixels) in the vertical direction (Z-axis direction) of the image from the pixel where the vertex 630 of the bounding box is located to the pixel where the reference line 610 is located, converted into a length in real space, as the upward warping amount Δy of the steel material 100. Note that when upward warping occurs, the height of the tip 100a of the steel material 100 is the highest. Therefore, the calculation unit 260 may calculate the length (number of pixels) in the vertical direction (Z-axis direction) of the image from the pixel at the top position (the position closest to the positive side of the Z axis) of the bounding box 620 to the pixel where the reference line 610 is located, converted into a length in real space, as the upward warping amount Δy of the steel material 100. Note that the positions of the vertices of the bounding box 620 are calculated based on, for example, the center position of the bounding box and the length and width of the bounding box. The upward bowing amount Δy of the steel material 100 does not have to be the length (number of pixels) in the vertical direction (Z-axis direction) of the image from the pixel where the vertex 630 of the bounding box is located to the pixel where the reference line 610 is located, converted into a length in real space. For example, the length (number of pixels) in the vertical direction (Z-axis direction) of the image from the pixel where the vertex 630 of the bounding box is located to the pixel where the reference line 610 is located may be the upward bowing amount Δy of the steel material 100. FIG. 6 illustrates the upward bowing amount Δy of the steel material 100 in such a case.

[0081] Here, information that quantifies and represents the shape of an object (steel material 100 in this embodiment) along the longitudinal direction is exemplified. However, it is not necessary to quantify and represent the shape of an object (steel material 100 in this embodiment) along the longitudinal direction. For example, the shape of an object along the longitudinal direction does not have to be quantified and represented (i.e., it does not have to be numerical information corresponding to the shape). For example, the information that represents the shape of an object along the longitudinal direction may be information that qualitatively represents the shape of an object along the longitudinal direction. For example, the information that qualitatively represents the shape of an object along the longitudinal direction may be information that indicates which of a first shape, a second shape, ..., an nth shape, the shape of the object along the longitudinal direction is. The "first shape, second shape, ..., nth shape" may be, for example, an upward camber, a downward camber, or a sickle-shaped camber. In addition, the information qualitatively representing the shape of the object along the longitudinal direction may be information indicating whether the shape of the object along the longitudinal direction is a large upward camber, a medium upward camber, or a slight upward camber.

[0082] Information indicating the change over time in the position of the object (steel material 100 in this embodiment) is obtained by tracking the position of the object in each object photographing image (each frame image). The position of which part of the object to be tracked may be set in advance (the object position may be the position of a specified part of the object), or may be set automatically by the calculation unit 260. For example, the calculation unit 260 may identify a part of the object that deviates from a flat state in the object photographing image, and calculate information indicating the change over time in the position of the identified part. The object position may be the position of the front end, the rear end, or the middle part.

[0083] In this embodiment, a case will be illustrated in which the information indicating the change over time in the position of an object is information indicating the change over time in the position of the tip 100a of the steel material 100. In the following description, the information indicating the change over time in the position of the tip 100a of the steel material 100 will be referred to as the trajectory of the tip of the steel material as necessary.

[0084] Fig. 7 is a diagram for explaining an example of a method for calculating the path of the tip of a steel material, and Fig. 8 is a diagram showing an example of the path of the tip of a steel material. In FIG. 7, the object-capturing images 700a, 700b, and 700c are object-capturing images at times ta, tb, and tc. It is assumed that time tb is a time after time ta, and time tc is a time after time tb (that is, ta < tb < tc). Note that the "···" shown between the object-capturing images 700a, 700b, and 700c indicates that object-capturing images are also captured between the object-capturing images 700a, 700b, and 700c.

[0085] The calculation unit 260 calculates the position (representative value) of the tip 100a of the steel material 100 in each of the object-capturing images 700a to 700c, and calculates information indicating each calculated position as the locus 810 of the tip of the steel material. In the present embodiment, a case where the learned model is YOLO v5 will be exemplified. Therefore, for example, the calculation unit 260 specifies the pixels of the vertices 720a, 720b, and 720c on the downstream side (the positive X-axis direction) and the upper side (the positive Z-axis direction) in the conveyance direction of the steel material 100 among the four vertices of the bounding boxes 710a, 710b, and 710c. Then, the calculation unit 260 may calculate, as information indicating the positions of the specified vertices 720a, 720b, and 720c, for example, a line connecting two vertices that are temporally adjacent to each other among the specified vertices 720a, 720b, and 720c as the locus 810 of the tip of the steel material. Further, regarding the position (Z coordinate) of the tip 100a of the steel material 100 in the vertical direction of the image as a function of time, based on the positions of the vertices 720a, 720b, and 720c and the times ta, tb, and tc, the coefficients and constants (intercepts) of the function may be calculated by the least squares method or the like, and the function having the coefficients and constants may be calculated as the locus 810 of the tip of the steel material. Further, as shown in FIG. 8, the calculation unit 260 may generate image data in which the locus 810 of the tip of the steel material is superimposed on the object-capturing image (for example, the last accumulated object-capturing image) 800 accumulated in the accumulation unit 250.

[0086] In YOLO v5, a reliability score indicating the reliability of the detection of the detected object is calculated for each detected object (bounding box). For bounding boxes whose reliability scores exceed a threshold, the calculation unit 260 calculates at least one of information that quantifies and represents the shape along the longitudinal direction of the object and information that indicates the change in the position of the object over time, and for bounding boxes whose reliability scores do not exceed a threshold, it is not necessary to calculate at least one of information that quantifies and represents the shape along the longitudinal direction of the object and information that indicates the change in the position of the object over time.

[0087] FIG. 9 is a diagram for explaining how the trajectory of the tip of the steel material 100 changes depending on the warp of the steel material. FIG. 9(a) is a diagram showing an example of a trajectory 900a of the tip of the steel material 100 when the steel material 100 has an upward camber. FIG. 9(b) is a diagram showing an example of a trajectory 900b of the tip of the steel material 100 when the steel material 100 has a downward camber or a sickle-shaped camber. As shown in FIG. 9(a), when the steel material 100 has an upward camber, the trajectory 900a of the tip of the steel material 100 (the height position of the tip 100a of the steel material 100) does not change significantly even after time t has passed (the steel material 100 is transported without moving up and down significantly). In contrast, as shown in FIG. 9(b), when the steel material 100 has a downward camber or a sickle-shaped camber, the trajectory 900b of the tip of the steel material 100 (the height position of the tip 100a of the steel material 100) changes significantly up and down as time t has passed (the steel material 100 is transported while moving up and down significantly). This corresponds to the fact that, when the steel material 100 has a downward or sickle-shaped bow, the tip 100a of the steel material 100 comes into contact with the conveying tables 120a to 120f, causing the steel material 100 to bounce while being conveyed. Therefore, by referring to the trajectories 900a, 900b of the steel material tip, it is possible to estimate the type of bow of the steel material 100. Based on the trajectory 900b of the steel material tip (for example, the difference between the maximum value and the minimum value), it is possible to estimate the degree of the bow. In this way, the object shape information (information relating to the shape of an object) may be information that allows the shape of an object to be evaluated, and may be information that contributes to estimating the shape of an object.

[0088] <Output unit 270> The output unit 270 outputs the object shape information calculated by the calculation unit 260. The output unit 270 may also output at least one (at least one frame image) of the object photographing images stored by the storage unit 250. The destination of the information output by the output unit 270 may be an operation management device. In this case, the operation management device may, for example, store the identification information (steel product number) of the steel product 100 transmitted to the processing device 200 and the object shape information and the object photographing image received from the processing device 200 (output unit 270) in association with each other. For example, the object shape information and the object photographing image to be associated with the identification information (steel product number) of the steel product 100 may be specified based on the transmission time of the identification information (steel product number) of the steel product 100 and the reception time of the object shape information and the object photographing image. Such association may also be performed by the processing device 200. In addition, the form of information output by the output unit 270 may be display on a computer display, storage in a storage medium inside or outside the processing device 200, or transmission to an external information processing device other than the operation management device.

[0089] [flowchart] Next, an example of a processing method using the processing device 200 of this embodiment will be described with reference to the flowcharts of Figures 10A and 10B. For ease of explanation, it is assumed that information that needs to be acquired in advance for the processing device 200 to calculate object shape information, in addition to the information clearly shown in the flowcharts of Figures 10A and 10B, is acquired by the acquisition unit 210 before the processing according to the flowcharts of Figures 10A or 10B starts.

[0090] In step S1001 of FIG. 10A, the acquisition unit 210 acquires the object photographing images 300a-300b. The object photographing images 300a-300b are used to determine whether or not there is an object being transported through the photographing area of ​​the photographing device 140 (in this embodiment, whether or not the image photographed by the photographing device 140 includes an object). Here, a case where one object photographing image (frame image) is acquired in step S1001 is illustrated. The cycle of the repetition of the processes of steps S1001-S1004 is, for example, the same time as the reciprocal of the frame rate of the photographing device 140. The object photographing image (frame image) acquired in step S1001 is not limited to one. For example, the processes of steps S1001-S1004 may be parallel processes for a plurality of object photographing images.

[0091] Next, in step S1002, the image processing unit 220 performs image processing on the object photographing images 300a to 300b. In this embodiment, a case is illustrated in which the blob area in the upstream area 410 and the blob area in the downstream area 420 are obtained as a result of the image processing in step S1002 (see FIG. 4). Details of step S1002 will be described later with reference to FIG. 10B.

[0092] Next, in step S1003, the object presence / absence determination unit 230 determines whether or not there is an object being transported through the imaging area of ​​the imaging device 140. In this embodiment, an example is shown in which if the blob area in the upstream area 410 is within a predetermined range defined for the upstream area 410 and the blob area in the downstream area 420 is within a predetermined range defined for the downstream area 420, it is determined that there is an object being transported through the imaging area of ​​the imaging device 140, and if not, it is determined that there is no object being transported through the imaging area of ​​the imaging device 140.

[0093] If the result of the determination in step S1003 is that there is no object being transported through the shooting area of ​​the image capturing device 140 (NO in step S1003), the process of step S1001 is performed again. On the other hand, if the result of the determination in step S1003 is that there is an object being transported through the shooting area of ​​the image capturing device 140 (YES in step S1003), the process of step S1004 is performed.

[0094] In step S1004, the calculation start condition determination unit 240 determines whether or not the calculation start condition for the object shape information is satisfied. In this embodiment, a case is illustrated in which it is determined that the calculation start condition is satisfied when the object identification information (steel product number) is (newly) acquired by the acquisition unit 210 and a certain time has passed since the object identification information (steel product number) was acquired.

[0095] If the result of the determination in step S1004 is that the calculation start condition for object shape information is not satisfied (NO in step S1004), the process of step S1001 is performed again. On the other hand, if the result of the determination in step S1004 is that the calculation start condition for object shape information is satisfied (YES in step S1003), the process of step S1005 is performed.

[0096] In step S1005, accumulation unit 250 accumulates object photographing image 600 acquired by acquisition unit 210 at a timing after the timing at which it is determined in step S1004 that the calculation start condition for object shape information is satisfied. Object photographing image 600 is used to calculate object shape information.

[0097] Next, in step S1006, the storage unit 250 judges whether or not the object photographing images 600 have been stored for a predetermined accumulation time. For example, the storage unit 250 judges whether or not the time counted from the timing at which it is judged in step S1004 that the calculation start condition for object shape information is satisfied is equal to or longer than the predetermined accumulation time. If the result of this judgment is that the object photographing images 600 have not been stored for the accumulation time (NO in step S1006), the process of step S1005 is performed again.

[0098] On the other hand, when the object photographing images 600 have been accumulated for the accumulation time (YES in step S1006), the process of step S1007 is performed. In step S1007, the calculation unit 260 calculates object shape information based on the object photographing images 600 accumulated in step S1005. In this embodiment, a case is illustrated in which the calculation unit 260 calculates the upward bowing amount Δy of the steel material 100 and the trajectory 810 of the steel material tip end as the object shape information (see Figs. 6 and 8). Note that the upward bowing amount Δy of the steel material 100 is calculated, for example, for each object photographing image 600 (each frame image).

[0099] Next, in step S1008, output unit 270 outputs the object shape information calculated in step S1007. When the process of step S1008 ends, the process according to the flowchart in Fig. 10A ends.

[0100] Next, an example of the process of step S1002 in FIG. 10A (image processing by the image processing unit 220) will be described with reference to the flowchart in FIG. 10B.

[0101] First, in step S1021, image processing unit 220 converts the (one) object photographing image and the background image acquired in step S1001 in Fig. 10A into grayscale images. Note that, here, a case is illustrated in which the background image is acquired by acquisition unit 210 before the start of the flowchart in Fig. 10B.

[0102] Next, in step S1022, image processing unit 220 creates a difference image between the object photographing image and the background image that have been grayscaled in step S1021. Next, in step S1023, the image processing unit 220 smoothes the background image created in step S1022.

[0103] Next, in step S1024, the image processing unit 220 creates a binary image by binarizing each pixel value of the smoothed difference image. Next, in step S1025, the image processing unit 220 performs noise reduction processing to reduce noise contained in the binarized image.

[0104] Next, in step S1026, the image processing unit 220 extracts the upstream area 410 of the noise-reduced binarized image 400 (see FIG. 4). Next, in step S1027, the image processing unit 220 calculates the area of ​​the blob in the upstream area 410.

[0105] Next, in step S1028, the image processing unit 220 extracts the downstream area 420 of the noise-reduced binarized image 400 (see FIG. 4). Next, in step S1029, the image processing unit 220 calculates the area of ​​the blob in the downstream area 420. When the process of step S1029 ends, the process according to the flowchart in Fig. 10B ends. Then, the process of step S1003 in Fig. 10A is performed.

[0106] [summary] As described above, in this embodiment, when the processing device 200 determines that there is a steel material 100 being transported through the photographing area of ​​the photographing device 140, the processing device 200 calculates object shape information, which is information about the shape of the steel material 100, using a trained model that has learned the relationship between an image photographed of the photographing area of ​​the photographing device 140 and the object shape information, based on an object photographing image photographed at a timing after the timing of the determination. Therefore, it is possible to prevent the calculation of object shape information using the trained model based on an object photographing image that does not include the steel material 100. Therefore, it is possible to achieve both high accuracy in the calculation of object shape information by using the trained model and suppression of time delays in the calculation of object shape information by using the trained model.

[0107] Furthermore, in this embodiment, the processing device 200 judges whether or not there is a steel material 100 being transported through the shooting area of ​​the camera 140, depending on whether or not the steel material 100 is included in the object shooting images 300a, 300b. Therefore, it is possible to judge whether or not there is a steel material 100 being transported through the shooting area of ​​the camera 140 in a short cycle (for example, the cycle of the frame rate of the camera 140) and with high accuracy.

[0108] Furthermore, in this embodiment, the processing device 200 judges whether or not the object photographing images 300a, 300b contain steel materials 100 based on the results of image processing on the object photographing images 300a, 300b. Therefore, it is possible to clearly distinguish the area of ​​the steel materials 100 from the other areas in the object photographing images 300a, 300b. Therefore, it is possible to judge with higher accuracy whether or not there is steel material 100 being transported in the photographing area of ​​the photographing device 140.

[0109] Furthermore, in this embodiment, the processing device 200 performs image processing for each of a plurality of regions (for example, the upstream area 410 and the downstream area 420) obtained by dividing the image for photographing an object (binary image 400), and judges whether or not the image for photographing an object (binary image 400) contains steel material 100 based on the results of the image processing for each of the plurality of regions. Therefore, it is possible to prevent the accuracy of judging whether or not the image for photographing an object (binary image 400) contains steel material 100 from decreasing due to the environment around the steel material 100, without performing complicated processing.

[0110] Moreover, in this embodiment, the processing device 200 calculates the object shape information when the calculation start condition for the object shape information is satisfied. Therefore, it is possible to prevent the object shape information from being calculated at an inappropriate timing. For example, by including in the calculation start condition for the object shape information a condition under which the steel material 100 transported immediately before the steel material 100 for which the object shape information is to be calculated can be regarded as being transported downstream of the photographing area of ​​the photographing device 140, it is possible to prevent the object shape information calculated based on the object photographing image of the steel material 100 transported immediately before the steel material 100 identified by the object identification information (steel material No.) from being treated as the object shape information of the steel material 100 identified by the object identification information (steel material No.).

[0111] (Other embodiments) The above-described embodiment of the present invention can be realized by a computer executing a program. A computer-readable recording medium having the program recorded thereon and a computer program product such as the program can also be applied as an embodiment of the present invention. Examples of the recording medium that can be used include a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, and a ROM. The embodiment of the present invention can be realized by a PLC (Programmable Logic Controller) or dedicated hardware such as an ASIC (Application Specific Integrated Circuit). Furthermore, the above-described embodiments of the present invention are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0112] The disclosure of the above embodiment can be, for example, as follows. [Disclosure 1] an acquisition unit that acquires an image captured in an area including a transport path for an object, the image capture area being an area in which at least a part of a side surface of the object can be captured; an object presence / absence determination unit that determines whether or not the object is present during transportation to the photographing area; a calculation unit that calculates object shape information, which is information about a shape of the object, based on the image captured at a timing after the timing at which the object presence / absence determination unit determines that the object is present in the shooting area; and Equipped with The processing device, wherein the calculation unit calculates the object shape information using a trained model that has learned a relationship between an image captured of the shooting area and the object shape information. [Disclosure 2] The processing device according to Disclosure 1, wherein the object presence / absence determination unit determines whether or not the object is being transported through the shooting area based on whether or not the object is included in the image. [Disclosure 3] An image processing unit that performs image processing on the image, The processing device according to Disclosure 2, wherein the object presence / absence determination unit determines whether or not the object is included in the image based on a result of the image processing. [Disclosure 4] the image processing unit performs the image processing for each of a plurality of regions obtained by dividing the image, The processing device according to Disclosure 3, wherein the object presence / absence determination unit determines whether or not the object is included in the image based on a result of the image processing for each of the plurality of regions. [Disclosure 5] The processing apparatus according to Disclosure 4, wherein the plurality of regions include an upstream region and a downstream region in a transport direction of the object. [Disclosure 6] a calculation start condition determination unit that determines whether or not a calculation start condition for the object shape information is satisfied; The processing device according to any one of claims 1 to 5, wherein the calculation section calculates the object shape information when the calculation start condition determination section determines that the calculation start condition is satisfied. [Disclosure 7] The processing device according to Disclosure 6, wherein the calculation start condition includes a condition in which the object transported immediately before the object for which the object shape information is to be calculated can be regarded as being transported downstream of the shooting area. [Disclosure 8] The processing device according to any one of claims 1 to 7, wherein the object shape information includes at least one of information representing a shape of the object along a longitudinal direction and information representing a change in a position of the object over time. [Disclosure 9] The processing apparatus according to any one of Disclosures 1 to 8, wherein the photographing area is an area capable of photographing an end portion of a side surface of the object that is downstream in a transport direction of the object. [Disclosure 10] an acquiring step of acquiring an image captured in an area including a transport path of the object, the image capturing area being an area capable of capturing an image of at least a part of a side surface of the object; an object presence / absence determination step of determining whether or not the object is present during transportation in the photographing area; a calculation step of calculating object shape information, which is information about a shape of the object, based on the image captured at a timing after the timing at which it is determined that the object is present in the shooting area by the object presence / absence determination step; Equipped with A processing method in which the calculation step calculates the object shape information using a trained model that has learned the relationship between an image captured of the shooting area and the object shape information. [Disclosure 11] A program for causing a computer to function as each unit of the processing device according to any one of claims 1 to 9. [Explanation of symbols]

[0113] 100 steel 100a Tip 100b rear end 100c~100d side 110 Rolling Mill 110a~110b Work roll 120a~120f Transport table 130 Transport Path 140 Imaging Device 150 Sensors 200 Processing equipment 210 Acquisition Department 220 Image Processing Unit 230 Object presence / absence determination unit 240 Calculation start condition determination section 250 Storage Unit 260 Calculation Unit 270 Output section 300a~300b Images for object photography 400 binary images 410 Upstream Area 420 Downstream Area 600 Object Shooting Images 610 Reference Line 620 Bounding Box 630 bounding box vertices 700a~700c Object photography images 710a~710c Bounding Box 720a~720c Bounding box vertices 800 Object Shooting Images 810 Trajectory of steel tip 900a~900b Trajectory of steel tip t1~t7 Time of taking the image for photographing the object (acquisition time) Δt1 Processing time per image in the image processing unit Δt2 Processing time per image in the calculation section Δy: Amount of upward warping of steel

Claims

1. an acquisition unit that acquires an image captured in an area including a transport path for an object, the image capture area being an area in which at least a part of a side surface of the object can be captured; an object presence / absence determination unit that determines whether or not the object is present during transportation to the photographing area; a calculation unit that calculates object shape information, which is information about a shape of the object, based on the image captured at a timing after the timing at which the object presence / absence determination unit determines that the object is present in the shooting area; and Equipped with The calculation unit calculates the object shape information using a trained model that has learned a relationship between an image of the shooting area and the object shape information.

2. The processing device according to claim 1 , wherein the object presence / absence determining unit determines whether or not the object being transported through the photographing area is present based on whether or not the object is included in the image.

3. An image processing unit that performs image processing on the image, The processing device according to claim 2 , wherein the object presence / absence determining unit determines whether or not the object is included in the image based on a result of the image processing.

4. the image processing unit performs the image processing for each of a plurality of regions obtained by dividing the image, The processing device according to claim 3 , wherein the object presence / absence determining unit determines whether or not the object is included in the image based on a result of the image processing for each of the plurality of regions.

5. The processing apparatus according to claim 4 , wherein the plurality of regions include an upstream region and a downstream region in a transport direction of the object.

6. a calculation start condition determination unit that determines whether or not a calculation start condition for the object shape information is satisfied; 6. The processing device according to claim 1, wherein the calculation section calculates the object shape information when the calculation start condition determination section determines that the calculation start condition is satisfied.

7. The processing device according to claim 6 , wherein the calculation start condition includes a condition in which the object transported immediately before the object for which the object shape information is to be calculated can be regarded as being transported downstream of the imaging region.

8. The processing device according to claim 1 , wherein the object shape information includes at least one of information representing a shape of the object along a longitudinal direction and information indicating a change in position of the object over time.

9. 6. The processing apparatus according to claim 1, wherein the photographing area is an area capable of photographing an end portion of a side surface of the object that is downstream in a conveying direction of the object.

10. an acquiring step of acquiring an image captured in an area including a transport path of the object, the image capturing area being an area capable of capturing an image of at least a part of a side surface of the object; an object presence / absence determination step of determining whether or not the object is present during transportation in the photographing area; a calculation step of calculating object shape information, which is information about a shape of the object, based on the image captured at a timing after the timing at which it is determined that the object is present in the shooting area by the object presence / absence determination step; Equipped with A processing method in which the calculation step calculates the object shape information using a trained model that has learned the relationship between an image captured of the shooting area and the object shape information.

11. A program for causing a computer to function as each part of the processing device according to any one of claims 1 to 5.

Images