Inspection equipment

Abstract

To judge the degree of opening of a booklet easily and accurately.SOLUTION: Inspection equipment comprises: an image acquisition unit for acquiring a spread image of a booklet captured in an open state; a determination image generation unit for generating a determination image with a determination range including a gutter, which is the seam of a booklet, cropped from the spread image acquired by the image acquisition unit; and a state determination unit for determining the degree of opening of the booklet on the basis of at least one of the shape of white space around the gutter shown in the determination image and the state of the page faces shown in the determination image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an inspection device. [Background technology]

[0002] In recent years, in order to preserve paper documents as assets for future generations and to make them easily shareable and usable, paper books, magazines, and the like have been converted into electronic data. Specifically, Patent Document 1 discloses a technology for converting books and the like into electronic data using an image scanner (stand-type image scanner) that captures an image of an open book or the like from directly above.

[0003] Here, if a book or the like (hereinafter also referred to as a booklet) is not opened properly, the characters may be hidden or distorted. Therefore, when imaging for conversion to electronic data, it is better to be able to check whether the booklet is opened properly. If a person visually checks all pages, it takes a huge amount of time and work, and the probability of human error, such as oversight, increases. As a countermeasure to this, it is possible to determine whether the booklet is opened properly from the image. For example, Patent Document 2 discloses a technology for determining whether a forged product has any defects based on an image of a cast product. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2006-279828 A [Patent Document 2] JP 2023-076837 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 2 is for inspecting objects such as castings, and inspects the entire image or all divided regions. Since the appearance of a booklet varies greatly from page to page, the feature values ​​extracted from the entire image or divided regions include many feature values, such as layout features, that are unnecessary for determining the degree to which the booklet is opened, and there is a risk that such unnecessary feature values ​​will become noise and make it difficult to make an appropriate determination.

[0006] The present invention has been made in consideration of the above-mentioned points, and has an object to easily and accurately determine the degree to which a booklet is opened. [Means for solving the problem]

[0007] [1] One aspect of the present invention is an inspection device that includes an image acquisition unit that acquires a double-page spread image of a booklet with the pages open; a judgment image generation unit that generates a judgment image by cutting out a judgment range that includes a gutter, which is the binding part of the booklet, from the double-page spread image acquired by the image acquisition unit; and a state determination unit that determines the degree of opening of the booklet based on at least one of the shape of a margin around the gutter shown in the determination image and the state of the page shown in the determination image.

[0008] [2] Also, in one aspect of the present invention, in the inspection device described in [1], the judgment image generation unit estimates the gutter of the booklet from the double-page spread image using a trained model that has been subjected to machine learning to estimate the gutter, and generates the judgment image by cutting out a predetermined range from the estimated gutter in the direction of the fore-edge of the booklet.

[0009] [3] Moreover, in one aspect of the present invention, in the inspection device described in [1], the judgment image generation unit estimates the judgment range using a trained model that has undergone machine learning to estimate at least one of the margins around the gutter and the layout surface from the double-page spread image, and generates the judgment image by cutting out the estimated judgment range.

[0010] [4] Also, one aspect of the present invention is that in the inspection device described in [1], the judgment image generation unit generates the judgment image by cutting out a predetermined range from the double-page spread image.

[0011] [5] Furthermore, in one aspect of the present invention, in the inspection device described in [4], the preset range is a range having a predetermined width in the left-right direction from a central vertical line that divides the double-page spread image in half left and right.

[0012] [6] Also, one aspect of the present invention is that in the inspection device described in [1], the judgment image generation unit generates the judgment image by cutting out an image of a range determined by a user from the double-page spread image.

[0013] [7] In another aspect of the present invention, the inspection device described in any one of [1] to [6] further includes a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of an imaging device from a sensor of the imaging device that captures the double-page spread image, and a physical information reflection unit that corrects the double-page spread image so that it becomes an image captured from the front of the booklet, based on the physical information acquired by the physical information acquisition unit.

[0014] [8] In another aspect of the present invention, the inspection device described in any one of [1] to [6] further includes a physical information acquisition unit that acquires physical information from a sensor attached to an imaging device that captures the double-page spread image, the physical information including information regarding at least one of an orientation and a position of the imaging device, and a physical information reflection unit that corrects the double-page spread image so that it becomes an image captured from the front of the booklet, based on the physical information acquired by the physical information acquisition unit.

[0015] [9] In another aspect of the present invention, the inspection device described in any one of [1] to [6] further includes a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of the imaging stand or the booklet from a sensor attached to an imaging stand on which the booklet is placed, and a physical information reflection unit that corrects the double-page spread image so that it becomes an image captured from the front of the booklet, based on the physical information acquired by the physical information acquisition unit.

[0016]

[10] In another aspect of the present invention, the inspection device described in any one of [1] to [6] further includes a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of an imaging device, an imaging stand, or a booklet that captures the double-page spread image, based on a value input by a user, and a physical information reflection unit that corrects the double-page spread image to an image captured from the front of the booklet, based on the physical information acquired by the physical information acquisition unit.

[0017]

[11] Moreover, one aspect of the present invention is an inspection device according to any one of [1] to [6], wherein the judgment by the state judgment unit is performed by a trained model that has undergone machine learning to judge the degree of opening of the booklet based on at least one of the shape of the margin around the gutter shown in the judgment image and the state of the printing surface shown in the judgment image.

[0018]

[12] Moreover, according to one aspect of the present invention, in the inspection device described in any one of [1] to [6], the judgment by the state judgment unit is performed by a convolutional neural network (CNN) that has performed machine learning to judge the degree of opening of the booklet based on at least one of the shape of the margin around the gutter shown in the judgment image and the state of the printing surface shown in the judgment image.

[0019]

[13] Furthermore, one aspect of the present invention is the inspection device described in

[12] , wherein the convolutional neural network (CNN) is provided with global average pooling (GAP) in a final layer. Effect of the Invention

[0020] According to the present invention, the degree to which a booklet is opened can be determined simply and accurately. [Brief description of the drawings]

[0021] [Figure 1] FIG. 2 is a diagram for explaining the names of each part of a booklet. [Diagram 2] FIG. 2 is a functional configuration diagram showing an example of a functional configuration of an inspection apparatus according to an embodiment. [Diagram 3] 11 is a diagram for explaining a case where the imaging device captures an image of a booklet from the front. FIG. [Figure 4] FIG. 1 is a diagram for explaining global average pooling (GAP). [Diagram 5] 13A and 13B are diagrams for explaining a case in which the open state of the booklet is determined to be "good." [Figure 6] FIG. 11 is a first diagram for explaining a case in which the open state of the booklet is determined to be “bad”. [Figure 7] FIG. 2 is a second diagram for explaining a case where the open state of the booklet is determined to be "bad." [Figure 8] FIG. 4 is a sequence diagram for explaining an example of processing of the inspection apparatus according to the embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of hardware constituting an inspection apparatus according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] [overview] A booklet created by binding multiple sheets of paper has a seam (gutter). When the booklet has a large number of pages, the gutter is difficult to see when the booklet is open, making it difficult to open the booklet properly. Therefore, it is considered that the appearance of the paper surface around the gutter in the captured image will be different between an image of a properly opened booklet and an image of a booklet that is not properly opened near the gutter.

[0023] Based on this viewpoint, in this embodiment, the degree of opening of the booklet is judged using an image (a judgment image to be described later) obtained by cutting out a portion that clearly indicates whether the degree of opening of the booklet is appropriate or not, that is, a shape around the gutter. By judging the degree of opening of the booklet using a judgment image cut out from an image, it is possible to extract a feature amount required for judging the degree of opening of the booklet. More specifically, it is possible to easily and accurately judge the degree of opening of the booklet by focusing on the shape of the margin around the gutter shown in the image, for example, the shape of the margin from the gutter of the booklet to the printing surface, the size of the width of the margin from the gutter of the booklet to the printing surface shown in the image, the inclination of the pattern and characters written on the printing surface, etc.

[0024] [Embodiment] The inspection device according to the present embodiment will be described in detail below with reference to the accompanying drawings, showing preferred embodiments. Note that the present embodiment is not limited to these embodiments, and includes various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be appropriately combined. In addition, the components of the present embodiment can be omitted, replaced, or modified in various ways without departing from the scope of the present invention.

[0025] FIG. 1 is a diagram for explaining the names of each part of a booklet. FIG. 1 shows a schematic diagram of an open booklet viewed from above. A booklet 2 is a bundle of paper on which pictures, letters, etc. are written or printed, bound together by a predetermined method. Examples of booklets 2 include books and magazines. In the following description, the stitch that binds the bundle of paper constituting the booklet 2 is referred to as the gutter 6, and the part corresponding to the cut surface of the book that faces the spine when the book is closed is referred to as the fore-edge. In addition, on the paper surface that can be seen when the booklet 2 is opened, the surface on which pictures, letters, etc. are written is referred to as the printing surface 8, the margin from the gutter 6 to the printing surface 8 is referred to as the gutter gap 7, and the margin from the fore-edge side to the printing surface 8 (the part that is touched when turning the page) is referred to as the fore-edge gap 9. Furthermore, the margin from the gutter 6 to the printing surface 8 of the booklet 2 that is captured in the captured image is referred to as the captured margin. The determination range JR will be described later.

[0026] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of the inspection device according to the embodiment. The inspection device 1 includes an operation receiving unit 21 and a control unit 22 in a processor 11. The control unit 22 further includes an image acquisition unit 23, a physical information acquisition unit 24, a physical information reflection unit 25, a judgment image generation unit 26, and a state judgment unit 27. Here, the processor 11 is, for example, a CPU (Central Processing Unit), and executes a judgment process (hereinafter, may be referred to as a judgment process) for judging the degree of opening of the booklet 2, thereby realizing each functional unit of the inspection device 1. In addition, each functional unit may be realized by using an electronic circuit as necessary. Furthermore, each functional unit does not have to be included in a single device, and the inspection device 1 may be configured by a plurality of devices.

[0027] The inspection device 1 is operated by a user. The user is, for example, a person who performs the task of converting the booklet 2 into electronic data. The user operates the mouse 151 and keyboard 152 to instruct the inspection device 1 to start a judgment process.

[0028] The operation reception unit 21 receives operations from a user. The operation reception unit 21 acquires operation information OI from an operation device such as a mouse 151 or a keyboard 152. The operation information OI includes, for example, information on an operation by a user using the operation device. The operation reception unit 21 transmits the acquired operation information OI to the control unit 22.

[0029] The imaging device 4 captures an image of the booklet 2 with the pages open. The image acquisition unit 23 acquires an image of the booklet 2 (a double-page spread image FSIM) from the imaging device 4. The imaging device 4 may be, for example, a CCD camera using a CCD (Charge Coupled Devices) image sensor, or a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging device 4 may also be a terminal device such as a smartphone, or a stand-type image scanner.

[0030] The sensor 5 acquires information (physical information PI) about the situation in which the imaging device 4 captures the booklet 2. The physical information PI includes, for example, information about the position where the imaging device 4 is installed, the direction in which the imaging device 4 faces, the angle between the surface of the imaging stand 3 on which the booklet 2 is placed and the imaging direction of the imaging device 4, the inclination of the imaging stand 3, and the like. The physical information acquisition unit 24 acquires the physical information PI from the sensor 5. The sensor 5 may be provided inside the imaging device 4, or may be attached to the imaging device 4 or the imaging stand 3. The sensor 5 may also be, for example, an inclination sensor or an acceleration sensor capable of detecting the inclination of the imaging device 4. Furthermore, instead of the sensor 5, information about the positions of the imaging device 4 and the imaging stand 3 may be acquired by using a device (for example, a smartphone) that receives an RFID (Radio Frequency Identifier) ​​tag, a beacon, or Wi-Fi (registered trademark) or Bluetooth (registered trademark). Instead of the sensor 5, a user may input a measured value to the inspection device 1 using an operation device such as a mouse 151 or a keyboard 152.

[0031] Here, a situation in which the imaging device 4 images the booklet 2 will be described. FIG. 3 is a diagram for explaining a case in which the imaging device images the booklet 2 placed on the imaging table 3 in an open state from a direction perpendicular to the surface of the imaging table 3. The imaging device 4 images the booklet from a direction perpendicular to the surface of the imaging table 3 (hereinafter, sometimes referred to as from the front) so that the spine 6 of the booklet 2 is at the center of the angle of view. When the imaging device 4 images the booklet 2 from the front, characters and the like in the image are less likely to be deformed or distorted, and are easier to read. However, there are cases in which the imaging device 4 does not image the booklet 2 from the front. When the imaging device 4 does not image the booklet 2 from the front, that is, when the imaging device 4 images the booklet 2 from a direction inclined at a predetermined angle with respect to the direction perpendicular to the surface of the imaging table 3 (hereinafter, sometimes referred to as from an oblique angle), characters and the like in the image may be deformed or distorted. If an image of the booklet 2 captured obliquely is converted directly into electronic data, a reader of the booklet 2 that has been converted into electronic data may find it difficult to read the characters. Also, when performing character recognition on the captured image to convert the characters captured in the image into text, the conversion may not be performed accurately if the characters are deformed or distorted. Therefore, when the imaging device 4 captures the booklet 2 obliquely, it is preferable to correct the captured double-page spread image FSIM to an image that appears as if it were captured from the front.

[0032] The physical information reflecting unit 25 corrects the double-page spread image FSIM to an image captured from the front by performing projective transformation based on information (physical information PI) on the angle and position at which the imaging device 4 captures the booklet 2. The physical information reflecting unit 25 transmits a corrected image, which is the corrected double-page spread image FSIM, to the control unit 22.

[0033] The judgment image generating unit 26 generates an image (hereinafter, may be referred to as a judgment image) used in the state judgment unit 27 described later to judge the degree of opening of the booklet 2 captured by the imaging device 4. The judgment image may be, for example, an image in which a range including the captured margin and the plate surface 8 (hereinafter, may be referred to as a judgment range JR) is cut out so that the shape of the captured margin can be understood. The judgment image generating unit 26 determines the judgment range JR, which is the range in which the judgment image is cut out from the corrected image, by a predetermined method. The judgment image generating unit 26 generates a judgment image in which the image of the judgment range JR is cut out from the corrected image. The judgment image generating unit 26 transmits the judgment image to the control unit 22. Note that the judgment image generating unit 26 may cut out an image of a range including the captured margin and a part of the plate surface 8 near the captured margin as the judgment image from the corrected image. By capturing the plate surface 8 in the judgment image, the judgment image generating unit 26 can grasp the shape of the captured margin. Furthermore, a judgment image that is cut out so as to include a portion of the printing surface 8 can reduce data compared to a judgment image that is cut out so as to include the entire printing surface 8.

[0034] The predetermined method for determining the judgment range JR may be, for example, a method in which the location of the gutter 6 is estimated from the corrected image using a trained model, and a range of a predetermined width from the estimated location of the gutter 6 in the left-right direction (toward the fore-edge opening 9) is set as the judgment range JR. Specifically, first, the judgment image generating unit 26 uses the trained model to estimate the portion of the gutter 6 of the booklet 2 depicted in the corrected image from the corrected image. Next, the judgment image generating unit 26 sets a range of a predetermined width from the estimated portion of the gutter 6 as the judgment range JR. The predetermined width may be, for example, a value set in advance (e.g., 300 [pixels]). When the predetermined width is set in advance, the judgment image generating unit 26 does not require complicated processing or calculation methods, and can easily generate the judgment image. The predetermined width may also be a value that changes dynamically based on, for example, the ratio of the value of the number of pages currently opened to the value of the number of pages of the entire booklet 2. In a thick booklet 2, it is difficult to open the gutter 6 of a page near the center compared to a page near the cover due to the thickness of the paper, and the size of the captured margin is likely to be smaller than when an image of the booklet 2 is captured with the gutter 6 properly opened. If the predetermined width is determined based on the ratio of the value of the number of pages currently opened to the value of the number of pages of the entire booklet 2, the determination image generating unit 26 can generate a determination image including the captured margin and the printing surface 8 even if the size of the margin portion shown in the image changes depending on the page. Furthermore, the predetermined width may be, for example, a value based on a change in pixel value from the estimated gutter 6 location to the fore-edge space 9 direction. If the predetermined width is determined based on a change in pixel value from the estimated gutter 6 location to the fore-edge space 9 direction, the determination image generating unit 26 can generate a determination image including the captured margin and the printing surface 8 even if the size of the captured margin differs for each booklet 2 or page.

[0035] The predetermined method for determining the judgment range JR may be, for example, a method of estimating the judgment range JR including at least one of the captured margin and the page 8 from the corrected image using a trained model. That is, the judgment image generating unit 26 may estimate the captured margin and cut out a predetermined range in the vicinity thereof. The judgment image generating unit 26 may estimate the location of the page 8 and assume that the area between them is the captured margin, thereby cutting out a range including the captured margin and a part of the page 8. The captured margin and the page 8 tend to have a large difference in pixel value (for example, the page is black and the captured margin is white), so they are easy to distinguish. Therefore, the judgment image generating unit 26 estimates at least one of the captured margin and the page 8, and determines the judgment range JR based on the estimated captured margin or page 8, thereby making it possible to accurately determine the judgment range JR including the captured margin and the page 8. The judgment image generating unit 26 may directly estimate a range including the captured margin and the page 8 using a machine learning-based learning model.

[0036] Furthermore, the judgment image generating unit 26 may generate the judgment image by cutting out an image included in a preset judgment range JR from the corrected image. When a double-sided printed booklet 2 is opened, the plate surfaces 8 are present on both the left and right pages. When imaging a double-sided printed booklet 2 for converting it into electronic data, the image is generally captured so that the spine 6 of the booklet 2 is located at the center of the image in order to include both plate surfaces 8. The judgment image generating unit 26 can easily generate the judgment image without requiring complex processing by, for example, generating the judgment image by setting the judgment range JR as a range of a predetermined width in the left-right direction from a central vertical line that divides the corrected image into left and right equal parts.

[0037] Furthermore, the judgment range JR may be determined by the user for each corrected image or booklet 2 to be imaged. A top-bound, single-sided printed booklet 2 or a side-bound, single-sided printed booklet 2 is generally imaged so that one of the pages on which the plate surface 8 exists is located at the center of the angle of view of the imaging device 4. If the judgment range JR is set in advance to be near the center of the corrected image, the judgment image generating unit 26 cannot generate a judgment image that includes the captured margin and the plate surface 8. By the user determining the judgment range JR for each corrected image or booklet 2 to be imaged, the judgment image generating unit 26 can accurately generate a judgment image even if the captured margin is not located near the center of the corrected image.

[0038] When the booklet 2 is opened, the shape of the captured margin is different between when it is opened properly and when it is not opened properly. When the booklet 2 is not opened properly, the paper surface of the opened page is curved, causing characters and the like written on the plate 8 to tilt, or when the paper surface of one page is curved, the plate surface of the other page is hidden, causing characters and the like to become unreadable and cut off. The state determination unit 27 determines the degree of opening of the captured booklet 2 based on the shape of the captured margin shown in the determination image generated by the determination image generation unit 26 and the state of characters and the like written on the plate 8 (for example, whether characters and the like are tilted or not cut off, etc.). Here, the degree of opening of the booklet 2 may be expressed as a binary value of, for example, "good" and "bad". The degree of opening may be, for example, a value estimated by the state determination unit 27 of the probability that a problem (re-photographing) will occur due to insufficient opening of the booklet 2. The state determination unit 27 transmits to the control unit 22 information relating to the open state of the booklet 2 shown in the double-page spread image FSIM (hereinafter, may be referred to as determination information JI).

[0039] The state determination unit 27 may use a trained model that has undergone machine learning to determine the degree of opening of the booklet 2. The state determination unit 27 may also use, for example, a convolutional neural network (hereinafter, sometimes referred to as CNN) as the trained model. The CNN can extract features from an image by performing convolution using a convolution layer, and make a judgment that captures a local pattern or shape within the image. According to the trained model using the CNN, the state determination unit 27 can determine the degree of opening of the booklet 2 based on, for example, a difference in the shape of the captured margins, whether or not the characters written on the plate surface 8 are tilted, and other conditions.

[0040] The state determination unit 27 may include global average pooling (hereinafter, may be referred to as GAP) in the final layer of the CNN (for example, a layer near the output layer) as a learned model. FIG. 4 is a diagram for explaining global average pooling (GAP). With reference to FIG. 4, a case where a GAP is included in the final layer of the CNN and a case where a GAP is not included will be described. FIG. 4(A) is a diagram showing a case where a fully connected layer 202 is included in the final layer of the CNN. FIG. 4(B) is a diagram showing a case where a global average pooling 203 is included in the final layer of the CNN. For example, a specific description will be given of a case where a convolution is performed by the CNN on an input image with a pixel count of "1000×1000" to produce a feature amount map of "7×7" having 512 channels (hereinafter, may be referred to as convolution layer 201). Each of the “7×7” channels of the convolution layer 201 becomes an image (mosaic image) that captures the rough shape of an object appearing in the “1000×1000” input image.

[0041] When the fully connected layer 202 is provided in the final layer of the CNN, the state determination unit 27 connects all neurons (7×7×512 [pieces]) in the convolution layer 201 having 512 channels to all neurons (e.g., 64×64 [pieces]) in the fully connected layer 202 in the fully connected layer 202, and performs calculations (FIG. 4(A)). That is, when the fully connected layer 202 is provided in the final layer of the CNN, the state determination unit 27 uses all information in the convolution layer 201 to perform calculations (determinations) during inference. When the state determination unit 27 performs a determination using all information in the “7×7” convolution layer 201 (feature map) having 512 channels, information on the shape of an object appearing in an input image, among the information held by the convolution layer 201, may affect the determination result. Specifically, when machine learning is performed using learning data in which there is a cause point (hereinafter referred to as the cause point) on the left side of the judgment image that may cause the opening state to be “poor”, state judgment unit 27 may not be able to judge the opening state to be “poor” even if the cause point is on the right side of the judgment image.

[0042] On the other hand, when the final layer of the CNN is provided with the global average pooling 203, the state determination unit 27 calculates the average of each channel of "7x7" of the convolution layer 201 in the global average pooling 203 (FIG. 4(B)). That is, the global average pooling 203 converts the "7x7" data of each channel into "1x1". At this time, the "1x1" data does not retain information about the shape of the object appearing in the input image of "1000x1000" pixels. According to a trained model in which the final layer of the CNN is provided with the global average pooling 203 in which information about the shape of the object appearing in the input image does not remain, the state determination unit 27 can determine the degree of opening as "bad" if there is a cause part, regardless of the position of the cause part in the determination image.

[0043] The control unit 22 transmits control information CI to the trained model storage unit 12 as necessary. The control information CI broadly includes information for the control unit 22 to control the trained model storage unit 12. The control information CI may include, for example, information for controlling the trained model storage unit 12 to transmit to the control unit 22 a trained model on which machine learning has been performed in advance.

[0044] The trained model storage unit 12 may store in advance information on, for example, a trained model on which machine learning has been performed to perform projective transformation on a double-page spread image FSIM and generate a corrected image, a trained model on which machine learning has been performed to estimate the portion of the gutter 6 from the corrected image, a trained model on which machine learning has been performed to estimate at least one of the imaging margin and the printing surface 8 from the corrected image, a trained model on which machine learning has been performed to estimate the judgment range JR from the corrected image, etc. The auxiliary storage device transmits trained model information LMI, which is information on the trained model, in response to the acquired control information CI.

[0045] The control unit 22 transmits the determination information JI to, for example, the display 153. That is, the inspection device 1 presents the determination result regarding the open state of the imaged booklet 2 to the user on the display 153. According to the determination information JI, the user may image the next page of the booklet 2 if the open state is "good", or may re-image the booklet 2 if the open state is "bad". Note that the inspection device 1 may present the open state of the booklet 2 to the user by, for example, a visual method such as a blinking light, an auditory method using sound, a tactile method such as a vibrating device, or other presentation methods.

[0046] Next, specific examples of the booklet 2 whose openness is judged to be "good" / "poor" will be described with reference to Figs. 5 to 7.

[0047] In order to prevent the characters and figures near the spine 6 from becoming difficult to read, a margin of a specified width (gutter gap 7) is often provided from the spine 6 to the page surface 8 on which the characters and figures are printed. The width of the gutter gap 7 is often the same for each page and each booklet. The shape of the gutter gap 7 is generally set to a rectangle.

[0048] In the case of an image captured of booklet 2 with the area around gutter 6 appropriately opened so that no deformation or distortion occurs in the characters or figures when viewed from directly above the printing surface 8, no deformation or distortion occurs in the characters or figures, and the shape of the margin portion from gutter 6 of booklet 2 to printing surface 8 shown in the captured image (captured margin) is often a preset shape, for example a rectangle. That is, in the case of booklet 2 with the area around gutter 6 appropriately opened, the shape of the captured margin will be the same as the gutter gap 7 that is preset in booklet 2. Also, because the area around gutter 6 is appropriately opened, imaging device 4 images gutter gap 7 of booklet 2 from the front, and the width of the captured margin is larger than when imaging booklet 2 with the area around gutter 6 not appropriately opened.

[0049] In the case of a booklet 2 in which the vicinity of the gutter 6 is not properly opened, the opened pages are curved, so that the characters are tilted relative to the gutter 6, that is, in the case of vertical writing, the direction in which the characters are read may not be parallel to the direction of the gutter 6. When the characters are tilted relative to the gutter 6, the shape of the margin portion (imaged margin) from the gutter 6 of the booklet 2 to the characters, etc., captured in the image may be, for example, a trapezoid. In other words, in the case of a booklet 2 in which the vicinity of the gutter 6 is not properly opened, the shape of the captured margin may differ from the same shape as the gutter gap 7 set in advance in the booklet 2. In addition, when the vicinity of the gutter 6 is not properly opened, the imaging device 4 captures the gutter gap 7 of the booklet 2 from an oblique angle, or the gutter gap 7 is hidden by the page, so that the width of the captured margin may be smaller than when an image of a booklet 2 in which the vicinity of the gutter 6 is properly opened is captured.

[0050] FIG. 5 is a diagram for explaining a case where the openness of a booklet is judged to be "good". FIG. 5(A) shows an image IM1 captured of a booklet 2 that is written vertically and whose openness is judged to be "good". FIG. 5(B) shows an image IM2 captured of a booklet 2 that is written horizontally and whose openness is judged to be "good". The captured margins included in the judgment range JR of images IM1 and IM2 have a rectangular shape. Furthermore, the text included in the judgment range JR of image IM1 is parallel to the spine 6. Furthermore, images IM1 and IM2 are symmetrical in the width from the spine 6 to the text written on the left and right pages 8.

[0051] Fig. 5(C) shows image IM3 captured of booklet 2 that has vertical writing, figures, and is judged to have a "good" open condition. The captured margins included in the judgment range JR of image IM3 are rectangular in shape. Furthermore, the edges of the text and figures included in the judgment range JR of image IM3 are parallel to the spine 6. Furthermore, the width from the spine 6 to the text written on the left and right pages 8 of image IM3 is symmetrical.

[0052] FIG. 5 (D) shows an image IM4 of a booklet 2 having a figure printed across the gutter 6 and determined to have a "good" degree of opening. Since the figure is printed across the gutter 6, the determination range JR of the image IM4 does not include the image margin. The booklet 2 captured in the image IM4 does not include the image margin, but it needs to be determined to have a "good" degree of opening. If an image of a booklet 2 that is determined to have a "good" degree of opening even without the image margin is used as learning data, the booklet 2 that does not have an image margin because of insufficient opening may also be determined to have a "good" degree of opening, and the accuracy of the determination by the state determination unit 27 may decrease. Therefore, when learning a trained model used for determination in the state determination unit 27, it is desirable not to use an image in which a figure or the like crosses the gutter 6 in the learning data.

[0053] FIG. 6 is a first diagram for explaining a case where the open state of a booklet is judged to be "poor." FIG. 6(A) shows an image IM5 captured of a booklet 2 that is written vertically and whose open state is judged to be "poor." The booklet 2 captured in image IM5 has both pages that are not open enough and the pages are bent, so the text is captured at an angle. The shape of the captured margin included in the judgment range JR of image IM5 is not rectangular because the text is tilted. Furthermore, the booklet 2 captured in image IM5 has both pages that are bent, so some of the text is cut off.

[0054] FIG. 6(B) shows image IM6 captured of booklet 2 that is written vertically and whose openness is determined to be "poor." One (left) page of booklet 2 captured in image IM6 is not fully open, and the page is bent, so the text on the left side is captured tilted. The shape of the captured margin included in the judgment range JR of image IM6 is not rectangular, because the text on the left side is tilted. Also, the left page of booklet 2 captured in image IM6 is bent, so part of the text on the left side is cut off. In image IM6, the right page is fully open, and the text on the right side is not tilted, but the text on the left side is tilted, so it is necessary to determine that the openness is "poor."

[0055] Fig. 6(C) shows image IM7 captured of booklet 2 with horizontal writing and determined to have a "poor" degree of opening. The booklet 2 captured in image IM7 has a rectangular shape with small captured margins. Also, in image IM7, the width from the spine 6 to the text written on the left and right pages 8 is not symmetrical. This shows that the booklet has not been opened properly.

[0056] FIG. 7 is a second diagram for explaining a case where the degree of opening of a booklet is judged to be "poor." FIG. 7(A) shows image IM8 captured of booklet 2 that is written vertically, has drawings, and is judged to be "poorly open." The booklet 2 captured in image IM8 has text and drawings that are not tilted, and the captured margins are rectangular in shape, but the captured margins are small. Also, in image IM8, the width from the spine 6 to the text written on the left and right pages 8 is not symmetrical. It can therefore be seen that the degree of opening is insufficient.

[0057] 7B shows image IM9 of booklet 2 that has an illustration printed across spine 6 and is judged to be “poorly opened.” The illustration included in judgment range JR of image IM9 is not fully opened, and is therefore disconnected and shifted diagonally between the right and left pages.

[0058] 8 is a sequence diagram for explaining an example of processing by the inspection device according to the embodiment, and an example of the flow of processing by the inspection device 1 will be explained with reference to the same diagram.

[0059] (Step S301) A user instructs the inspection device 1 to start a determination process using an operation device such as the mouse 151 or the keyboard 152. The operation reception unit 21 acquires operation information OI related to the user's operation. The operation reception unit 21 transmits the operation information OI to the control unit 22. The control unit 22 included in the inspection device 1 acquires information related to the instruction to start the determination process from the user.

[0060] (Step S302) The control unit 22 instructs the image acquisition unit 23 to acquire a double-page spread image FSIM.

[0061] (Step S303) In accordance with an instruction from the control unit 22, the image acquiring unit 23 acquires a double-page spread image FSIM from the imaging device 4. That is, the image acquiring unit 23 acquires an image of the booklet 2 in an open state.

[0062] (Step S304) The image acquisition unit transmits the acquired double-page spread image FSIM to the control unit 22. The control unit 22 acquires the double-page spread image FSIM from the image acquisition unit .

[0063] (Step S305) For example, if the imaging device 4 is not imaging the booklet 2 from the front, the control unit 22 instructs the physical information acquisition unit 24 to acquire the physical information PI.

[0064] (Step S306) In accordance with an instruction from the control unit 22, the physical information acquisition unit 24 acquires the physical information PI from the sensor 5. That is, the physical information acquisition unit 24 acquires information on the angle and position at which the imaging device 4 captures the image of the booklet 2.

[0065] (Step S307) The physical information acquiring unit 24 transmits the acquired physical information PI to the control unit 22. The control unit 22 acquires the physical information PI from the physical information acquiring unit 24.

[0066] (Step S308) The control unit 22 instructs the physical information reflecting unit 25 to correct the double-page spread image. Accordingly, the control unit 22 transmits the double-page spread image FSIM and the physical information PI to the physical information reflecting unit 25. The physical information reflecting unit 25 acquires the double-page spread image FSIM and the physical information PI from the control unit 22.

[0067] (Step S309) The physical information reflecting unit 25 performs projective transformation based on information (physical information PI) about the angle and position at which the imaging device 4 captures the booklet 2, thereby correcting the double-page spread image FSIM to an image captured from the front.

[0068] (Step S310) The physical information reflecting unit 25 transmits a corrected image, which is the corrected double-page spread image FSIM, to the control unit 22. The control unit 22 acquires the corrected image from the physical information reflecting unit 25.

[0069] (Step S311) The control unit 22 instructs the determination image generating unit 26 to generate a determination image. In response to this, the control unit 22 transmits a corrected image to the determination image generating unit 26. The determination image generating unit 26 acquires the corrected image from the control unit 22.

[0070] (Step S312) The judgment image generating unit 26 determines a judgment range JR, which is a range for cutting out a judgment image used to judge the degree of opening from the corrected image. The method for determining the judgment range JR may be based on a predetermined algorithm, or may be determined by the user.

[0071] (Step S313) The judgment image generating unit 26 generates a judgment image by cutting out an image within the determined judgment range JR from the corrected image. The judgment image is, for example, an image in which the shape of the captured margin can be known.

[0072] (Step S314) The determination image generating unit transmits the generated determination image to the control unit 22. The control unit 22 acquires the determination image from the determination image generating unit .

[0073] (Step S315) The control unit 22 instructs the state determination unit 27 to determine the open state of the imaged booklet 2. Accordingly, the control unit 22 transmits a determination image to the state determination unit 27. The state determination unit 27 acquires the determination image from the control unit 22.

[0074] (Step S316) Based on the acquired judgment image, the state judgment unit 27 judges the open state of the captured booklet 2. At this time, the judgment result may be "good" or "bad", or may be the probability that a problem (re-imaging) will occur due to the booklet 2 not being open enough.

[0075] (Step S317) The state determination unit 27 transmits information on the determination result (determination information JI) to the control unit 22. The control unit 22 acquires the information on the determination result from the state determination unit 27.

[0076] (Step S318) The control unit 22 presents the user with the determination information JI, which is information related to the determination result. According to the determination information JI, the user may image the next page of the booklet 2 if the opening condition is "good", or may image the next page again if the opening condition is "bad".

[0077] After receiving an instruction from the user to start the determination process, the inspection device 1 may repeat steps S302 to S318 until receiving an instruction (not shown) to end the determination process.

[0078] 9 is a diagram showing an example of hardware constituting an inspection device according to an embodiment. As shown in FIG. 9, the inspection device 1 includes a processor 11, a main memory device 13, a communication interface 14, an auxiliary memory device 15, an input / output device 16, and a bus 17.

[0079] The main storage device 13 is, for example, a RAM (Random Access Memory), and stores in advance programs that are read and executed by the processor 11.

[0080] The communication interface 14 is an interface circuit for communicating with other devices via a network. The network is a predetermined communication network configured using lines. An example of a network is the Internet. The predetermined communication network is an open network, and devices connected to the communication network can communicate with each other using a predetermined protocol. In the Internet layer of the communication network, communication is performed using the Internet Protocol (IP) or the like. In addition, the network includes, for example, a WAN and a LAN.

[0081] The auxiliary storage device 15 is, for example, a hard disk drive, a solid state drive, or a flash memory. The role of the trained model storage unit 12 is played by the auxiliary storage device 15. If the auxiliary storage device 15 does not contain information on a desired trained model, the auxiliary storage device 15 may exchange information with the main storage device 13 as necessary. In addition, the control unit 22 may exchange information with the main storage device 13 without going through the auxiliary storage device 15.

[0082] The input / output device 16 is, for example, an input / output port. For example, the mouse 151, keyboard 152, display 153, imaging device 4, and sensor 5 shown in Fig. 9 are connected to the input / output device 16. The imaging device 4 and the sensor 5 may exchange information with the communication interface 14 via a network.

[0083] The bus 17 connects the processor 11, the main memory device 13, the communication interface 14, the auxiliary memory device 15, and the input / output device 16 so that they can transmit and receive data to and from one another.

[0084] In this embodiment, an example is shown in which the imaging device 4 images the booklet 2 from an oblique angle. However, this embodiment is not limited to this example, and the imaging device 4 may image the booklet 2 from the front. When the imaging device 4 images the booklet 2 from the front, the inspection device 1 does not necessarily need to include the physical information acquisition unit 24 and the physical information reflection unit 25. In addition, when the inspection device 1 is not provided with the physical information acquisition unit 24 and the physical information reflection unit 25, the judgment image generation unit 26 may acquire the double-page spread image FSIM from the image acquisition unit 23.

[0085] [Summary of this embodiment] According to the above-described embodiment, the inspection device 1 includes an image acquisition unit 23 that acquires a double-page spread image FSIM of the booklet with the pages open, a judgment image generation unit 26 that generates a judgment image by cutting out a judgment range JR including the gutter 6, which is the binding part of the booklet 2, from the double-page spread image FSIM acquired by the image acquisition unit 23, and a judgment unit 28 that judges the degree of opening of the booklet 2 based on at least one of the shape of the margin part (captured margin) around the gutter 6 shown in the judgment image and the state of the plate surface 8 shown in the judgment image. The booklet 2 is often provided with a gutter gap 7 to prevent characters and figures near the gutter 6 from becoming difficult to read. In addition, since the width of the gutter gap 7 is uniform for each page and each booklet, the shape of the gutter gap 7 is generally rectangular. The shape of the gutter gap 7 (captured margin) shown in the image, the inclination of the text, etc. differ between a case where the booklet 2 is properly opened and a case where it is not opened. Therefore, the inspection device 1 of the embodiment can easily and accurately determine the degree of opening of a booklet based on the shape of the captured margin in the captured image, the width of the margin from the spine of the booklet 2 to the printing surface 8 in the image, the inclination of the text in the image, etc.

[0086] According to the above-described embodiment, in the inspection device 1, the determination image generating unit 26 estimates the gutter 6 of the booklet 2 from the corrected image (or the double-page spread image FSIM) using a trained model that has been subjected to machine learning to estimate the gutter 6 of the booklet 2, and generates a determination image by cutting out a predetermined range from the estimated gutter 6 in the fore-edge direction of the booklet 2. By cutting out a predetermined range from the estimated gutter 6, differences are unlikely to occur in the size of the cut-out image and the proportion of the plate surface 8 included in the cut-out image. The state determining unit 27 can determine the open state of the captured booklet 2 without being affected by the size of the image or the proportion of the plate surface 8 included in the determination image by using the determination image generated by cutting out a predetermined range from the estimated gutter 6. In other words, the determination accuracy of the state determining unit 27 is improved by using the determination image generated by cutting out a predetermined range from the estimated gutter 6. Therefore, the inspection device 1 according to the embodiment can easily and accurately determine the open state of the booklet 2.

[0087] According to the above-described embodiment, in the inspection device 1, the judgment image generating unit 26 estimates a judgment range using a trained model that has undergone machine learning to estimate at least one of the margin portion around the gutter 6 and the plate surface 8 from the corrected image (or the double-page spread image FSIM), and generates a judgment image by cutting out the estimated judgment range. The captured margin and the plate surface 8 are easily distinguished because there is a large difference in pixel value (for example, the plate surface is black and the captured margin is white). Therefore, according to the present embodiment, the judgment image generating unit 26 estimates at least one of the captured margin and the plate surface 8, and determines the judgment range JR based on the estimated captured margin or the plate surface 8, thereby generating a judgment image that includes the captured margin and the plate surface 8.

[0088] Furthermore, according to the above-described embodiment, in the inspection device 1, the judgment image generating unit 26 generates a judgment image by cutting out a preset range from within the corrected image (or the double-page spread image FSIM). The inspection device 1 according to this embodiment can easily generate a judgment image by cutting out a preset range without requiring complex processing.

[0089] According to the above-described embodiment, the preset range in the inspection device 1 is a range having a predetermined width in the left-right direction from the center vertical line that divides the corrected image (or the double-page spread image FSIM) into left and right halves. When a double-sided printed booklet 2 is opened, the plate surfaces 8 are present on both the left and right pages. When imaging a double-sided printed booklet 2 for conversion into electronic data, the image is generally captured so that the spine 6 of the booklet 2 is located at the center of the image in order to include both plate surfaces 8. The inspection device 1 according to this embodiment generates a judgment image by using a range of a predetermined width in the left-right direction from the center vertical line that divides the corrected image into left and right halves as the judgment range JR, thereby making it possible to easily generate a judgment image without requiring complex processing.

[0090] According to the above-described embodiment, in the inspection device 1, the judgment image generating unit 26 generates a judgment image by cutting out an image of a range determined by the user from the corrected image (or the double-page spread image FSIM). A top-bound or side-bound booklet 2 printed on one side is generally imaged so that one page on which the plate surface 8 exists is located at the center of the angle of view of the imaging device 4. If the judgment range JR is set in advance to be near the center of the corrected image, it is difficult for the judgment image generating unit 26 to generate a judgment image that includes the captured margin and the plate surface 8. The inspection device 1 according to this embodiment can accurately generate a judgment image by determining the judgment range JR for each corrected image or booklet 2 to be imaged, even if the captured margin is not located near the center of the corrected image.

[0091] According to the above-described embodiment, the inspection device 1 further includes a physical information acquisition unit 24 that acquires physical information PI including information on at least one of the orientation and the position of the imaging device 4 from the sensor 5 of the imaging device 4 that captures the double-page spread image FSIM, and a physical information reflection unit 25 that corrects the double-page spread image FSIM to an image captured from the front of the booklet 2 based on the physical information PI acquired by the physical information acquisition unit 24. When the booklet 2 is captured from an oblique angle, characters and the like appearing in the image may be deformed or distorted. By standardizing the images used for the determination to images captured from the front, differences in the determination results of the degree of opening of the booklet 2 are unlikely to occur between when the booklet 2 is captured from the front and when the booklet 2 is captured from an oblique angle. The inspection device 1 according to this embodiment can accurately determine the degree of opening of the booklet 2 by performing projective transformation based on information on the orientation and position of the imaging device 4.

[0092] According to the above-described embodiment, the inspection device 1 further includes a physical information acquisition unit 24 that acquires physical information PI including information on at least one of the orientation and the position of the imaging device 4 from the sensor 5 attached to the imaging device 4 that captures the double-page spread image FSIM, and a physical information reflection unit 25 that corrects the double-page spread image FSIM to an image captured from the front of the booklet 2 based on the physical information PI acquired by the physical information acquisition unit 24. By standardizing the images used for the judgment to images captured from the front, differences in the judgment results of the degree of opening of the booklet 2 are unlikely to occur between when the booklet 2 is imaged from the front and when the booklet 2 is imaged from an oblique angle. The inspection device 1 according to this embodiment can accurately judge the degree of opening of the booklet 2 by performing projective transformation based on information on the orientation and position of the imaging device 4.

[0093] According to the above-described embodiment, the inspection device 1 further includes a physical information acquisition unit 24 that acquires physical information PI including information on at least one of the orientation and the position of the imaging stand 3 or the booklet 2 from the sensor 5 attached to the imaging stand 3 on which the booklet 2 is placed, and a physical information reflection unit 25 that corrects the double-page spread image FSIM to an image captured from the front of the booklet 2 based on the physical information PI acquired by the physical information acquisition unit 24. By standardizing the images used for the judgment to images captured from the front, differences in the judgment results of the degree of opening of the booklet 2 are unlikely to occur between when the booklet 2 is imaged from the front and when the booklet 2 is imaged from an oblique angle. The inspection device 1 according to this embodiment can accurately judge the degree of opening of the booklet 2 by performing projective transformation based on information on the orientation and position of the imaging device 4.

[0094] According to the above-described embodiment, the inspection device 1 further includes a physical information acquisition unit 24 that acquires physical information PI including information on at least one of the orientation or position of the imaging device 4, the imaging stand 3, or the booklet 2 that captures the double-page spread image FSIM based on a value input by a user, and a physical information reflection unit 25 that corrects the double-page spread image FSIM to an image captured from the front of the booklet 2 based on the physical information PI acquired by the physical information acquisition unit 24. By standardizing the images used for the judgment to images captured from the front, differences in the judgment results of the degree of opening of the booklet 2 are unlikely to occur between when the booklet 2 is imaged from the front and when the booklet 2 is imaged from an oblique angle. The inspection device 1 according to this embodiment can accurately judge the degree of opening of the booklet 2 by performing projective transformation based on information on the orientation and position of the imaging device 4.

[0095] According to the above-described embodiment, in the inspection device 1, the state determination unit 27 performs the determination based on at least one of the shape of the margin (image margin) around the gutter 6 shown in the determination image and the state of the plate surface 8 shown in the determination image, using a trained model that has undergone machine learning to determine the degree of opening of the booklet 2. By performing the determination using a trained model that has undergone machine learning, it becomes unnecessary for a person to visually check the degree of opening of all pages. Furthermore, according to the trained model that has undergone machine learning to be able to determine the degree of opening as "good" or "bad," it is possible to perform the determination even if there are differences between booklets 2, such as the width of the gutter 7 and the size of the booklet 2. Therefore, the inspection device 1 according to this embodiment can easily and accurately determine the degree of opening of the booklet 2 by using a trained model in the state determination unit 27.

[0096] According to the above-described embodiment, in the inspection device 1, the state determination unit 27 performs a determination by a convolutional neural network (CNN) that performs machine learning to determine the degree of opening of the booklet 2 based on at least one of the shape of the margin around the gutter 6 shown in the determination image and the state of the plate surface 8 shown in the determination image. The CNN can extract features from an image by performing convolution using a convolution layer, and can make a determination by capturing local patterns and shapes in the image. The inspection device 1 according to the present embodiment can determine the degree of opening of the booklet 2 based on a partial difference in the shape of the captured margin, whether or not the characters written on the plate surface 8 are tilted, etc., by making a determination on the local patterns and shapes in the image.

[0097] According to the above-described embodiment, in the inspection device 1, the convolutional neural network (CNN) includes a global average pooling 203 in the final layer. For example, when convolution is performed by the CNN on an input image with a pixel count of "1000 x 1000", the feature map (convolutional layer 201) has information on the rough shape of an object appearing in the input image. When the state determination unit 27 performs a determination using all the information in the feature map (convolutional layer 201), information on the rough shape of an object appearing in the input image, among the information in the feature map, may affect the determination result. When the global average pooling 203 is included in the final layer of the CNN, the state determination unit 27 calculates the average of each channel of "7 x 7" in the convolutional layer 201 in the global average pooling 203. That is, the global average pooling 203 converts the "7 x 7" data of each channel into "1 x 1". In this case, the "1 x 1" data does not contain any information about the shape of the object appearing in the input image with the number of pixels of "1000 x 1000". The inspection device 1 according to this embodiment uses a trained model that includes a GAP in the final layer of the CNN, which does not contain any information about the shape of the object appearing in the input image, and can determine the degree of opening as "bad" if there is a cause, regardless of the position of the cause in the determination image.

[0098] In addition, all or part of the functions of each unit of the inspection device 1 in the above-mentioned embodiment may be realized by recording a program for realizing these functions in a computer-readable recording medium, reading the program recorded in the recording medium into a computer system, and executing it. Note that the "computer system" here includes hardware such as an OS and peripheral devices.

[0099] Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and recording units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above-mentioned program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0100] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes and the like are possible without departing from the scope of the present invention. In addition, the configurations described in the above-mentioned embodiments and examples may be combined. [Explanation of symbols]

[0101] 1...inspection device, 2...booklet, 3...imaging stand, 4...imaging device, 5...sensor, 6...gutter, 7...gutter clearance, 8...printing surface, 9...edge clearance, 11...processor, 12...model storage unit, 13...main storage unit, 14...communication interface, 15...auxiliary storage unit, 16...input / output device, 17...bus, 21...operation reception unit, 22...control unit, 23...image acquisition unit, 24...physical information acquisition unit, 25...physical information reflection unit, 26...judgment image generation unit, 27...state judgment unit, FSIM...spread image, JR...judgment range, CI...control information, JI...judgment information, LMI...model information, OI...operation information, PI...physical information, 201...convolution layer, 202...fully connected layer, 203...global average pooling

Claims

1. an image acquisition unit for acquiring a double-page spread image of a booklet captured with the pages open; a determination image generating unit that generates a determination image by cutting out a determination range including a gutter, which is a binding portion of the booklet, from the double-page spread image acquired by the image acquiring unit; a state determination unit that determines an open state of the booklet based on at least one of a shape of a margin portion around the gutter shown in the determination image and a state of a printing surface shown in the determination image; An inspection device comprising:

2. The determination image generating unit estimates the gutter of the booklet from the double-page spread image using a trained model that has been subjected to machine learning for estimating the gutter, and generates the determination image by cutting out a predetermined range from the estimated gutter in the fore-edge direction of the booklet.

2. The inspection device according to claim 1.

3. The determination image generating unit estimates the determination range using a trained model that has been subjected to machine learning to estimate at least one of the margin portion around the gutter and the printing surface from the double-page spread image, and generates the determination image by cutting out the estimated determination range.

2. The inspection device according to claim 1.

4. The determination image generating unit generates the determination image by cutting out a preset range from the double-page spread image.

2. The inspection device according to claim 1.

5. The preset range is a range having a predetermined width in the left-right direction from a vertical center line that divides the double-page spread image into two halves.

5. The inspection apparatus according to claim 4.

6. The determination image generating unit generates the determination image by cutting out an image of a range determined by a user from the double-page spread image.

2. The inspection device according to claim 1.

7. a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of an imaging device from a sensor of the imaging device that captures the double-page spread image; a physical information reflecting unit that corrects the double-page spread image based on the physical information acquired by the physical information acquiring unit so that the image is an image captured from the front of the booklet; The inspection device according to claim 1 , further comprising:

8. a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of an imaging device from a sensor attached to the imaging device that captures the double-page spread image; a physical information reflecting unit that corrects the double-page spread image based on the physical information acquired by the physical information acquiring unit so that the image is an image captured from the front of the booklet; The inspection device according to claim 1 , further comprising:

9. a physical information acquiring unit that acquires physical information including information regarding at least one of an orientation and a position of the imaging stand or the booklet from a sensor attached to the imaging stand on which the booklet is placed; a physical information reflecting unit that corrects the double-page spread image based on the physical information acquired by the physical information acquiring unit so that the image is an image captured from the front of the booklet; The inspection device according to claim 1 , further comprising:

10. a physical information acquisition unit that acquires physical information including information regarding at least one of an orientation and a position of an imaging device, an imaging stand, or a booklet that captures the double-page spread image based on a value input by a user; a physical information reflecting unit that corrects the double-page spread image based on the physical information acquired by the physical information acquiring unit so that the image is an image captured from the front of the booklet; The inspection device according to claim 1 , further comprising:

11. The inspection device according to any one of claims 1 to 6, wherein the judgment by the state judgment unit is performed by a trained model that has undergone machine learning to judge the degree of opening of the booklet based on at least one of the shape of the margin around the gutter shown in the judgment image and the state of the printing surface shown in the judgment image.

12. The determination by the state determination unit is performed by a convolutional neural network (CNN) that performs machine learning to determine the degree of opening of the booklet based on at least one of the shape of the margin around the gutter shown in the determination image and the state of the page shown in the determination image. The inspection device according to any one of claims 1 to 6.

13. The convolutional neural network (CNN) is equipped with global average pooling (GAP) in the final layer.

13. The inspection apparatus according to claim 12.

Images