Image processing device, control method thereof, and storage medium

Abstract

The present disclosure relates to an image processing device, a control method for the image processing device, and a storage medium. The device detects a plurality of regions from an overall image, and for each of the plurality of regions, determines whether the region is a candidate for a region including an object to be processed based on a determination reference value (first determination process), obtains a magnified image of the region that is determined to be a candidate for a region including the object in the first determination process, and determines whether the obtained magnified image is an image of the object (second determination process). The device identifies a region in the overall image corresponding to the magnified image that is determined not to be an image of the object in the second determination process, and performs control to update the determination reference value used in the first determination process based on the image information of the identified region in the overall image.

Description

Technical Field

[0001] Aspects of the embodiments relate to an image processing apparatus, a control method for the image processing apparatus, and a storage medium. Background Art

[0002] Currently, a system that can control the pan, tilt, and zoom of a camera installed at a remote location via a network from a terminal on the monitoring side is widely popular. U.S. Patent Application Publication No. 2017 / 0293788 discusses the following technology: In such a monitoring system, a barcode is roughly searched, and the detected barcode image is enlarged and photographed by performing pan, tilt, and zoom operations to obtain a barcode image with high resolution. In addition, U.S. Patent Application Publication No. 2012 / 0070086 discusses a technology of photographing an image including a plurality of reading objects, sequentially identifying the positions of the reading objects, and sequentially performing reading processing by focusing on the identified positions.

[0003] In such a monitoring system, a certain degree of over-detection is performed by setting a low judgment threshold for detection to obtain a desired image without omission. However, there is a problem that frequent such over-detection increases the load in subsequent processing steps. Therefore, the overall processing takes a long time. Summary of the Invention

[0004] According to an aspect of the embodiment, the apparatus includes: a detection unit configured to detect a plurality of regions from an image; a first judgment unit configured to perform a first judgment process on each of the plurality of regions to judge whether each of the plurality of regions is a candidate for a region including a specific object based on a judgment reference value stored in a storage unit; a control unit configured to control an imaging device to photograph a magnified image of a region among the plurality of regions that is judged to be a candidate for a region including the specific object in the first judgment process; a receiving unit configured to receive the photographed magnified image; a second judgment unit configured to perform a second judgment process on the magnified image to judge whether the magnified image includes the specific object; and an update control unit configured to identify a region corresponding to the magnified image that is judged not to include the specific object in the second judgment process, and update the stored judgment reference value based on the image information of the identified region.

[0005] Other features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0006] Figure 1 is a block diagram showing the overall configuration of an image processing system.

[0007] Figure 2It is a block diagram showing the hardware structure of an image processing device.

[0008] Figure 3 It is a diagram showing the external view of a network (NW) camera.

[0009] Figure 4 It is a block diagram showing the hardware structure of the NW camera.

[0010] Figure 5 It is a diagram showing a loaded package.

[0011] Figure 6 It is a flowchart showing the process of preparation processing.

[0012] Figure 7 It is a diagram showing a matching model.

[0013] Figure 8 It is a diagram showing an example of a setting screen.

[0014] Figure 9 It is a diagram showing an example of a test screen.

[0015] Figure 10 It is a diagram showing an example of a file.

[0016] Figure 11 It is a flowchart showing the process of overall processing according to the first exemplary embodiment.

[0017] Figure 12A and Figure 12B It is a diagram showing over-detection.

[0018] Figure 13 It is a flowchart showing the process of preparation processing.

[0019] Figure 14A and Figure 14B They are diagrams each showing an example of a setting screen.

[0020] Figure 15A and Figure 15B They are diagrams each showing an example of a setting screen.

[0021] Figure 16 It is a diagram showing an example of a captured image screen.

[0022] Figure 17 It is a diagram showing an example of a read image screen.

[0023] Figure 18A and Figure 18B They are diagrams each showing an example of a display screen.

[0024] Figure 19 It is a flowchart showing the process of overall processing according to the second exemplary embodiment. Detailed implementation manners

[0025] The exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0026] <Overall structure of the image processing system>

[0027] Figure 1 It is a block diagram showing the overall structure of the image processing system. The image processing system according to the first exemplary embodiment is a system that captures images of loaded packages delivered to a factory or the like, reads barcodes attached to each loaded package, and compares the barcodes with the pre-registered contents of the barcodes to check whether the loaded packages and the like have been delivered as planned. In addition, in this exemplary embodiment, it is assumed that labels are attached to each package included in the loaded packages that are the imaging objects, and barcodes are described in each label. The image processing system according to this exemplary embodiment sequentially reads and compares the barcodes described in each label attached to each package. Although the description will be given of the case where the object for the image processing device 100 to perform the reading and comparison processing is a barcode, the object for the reading and comparison processing is not limited to a barcode.

[0028] Other examples of the object for the reading processing may include numbers, strings composed of characters and symbols, and Quick Response (QR) codes.

[0029] The image processing system includes an image processing device 100, a network (NW) camera 110, a database (DB) 120, and a Power over Ethernet (PoE) hub 130. The image processing system further includes a programmable logic controller (PLC) 140 and a sensor 150.

[0030] The PoE hub 130 is connected to the image processing device 100, the NW camera 110, the DB 120, and the PLC 140, communicates with each unit, and supplies power to each unit. The contents of the barcodes described in each of the multiple labels attached to the corresponding packages among the multiple packages to be delivered are pre-registered in the DB 120. The PLC 140 controls the overall image processing system. The sensor 150 detects that each loaded package has been delivered to a predetermined position.

[0031] The image processing device 100 is connected to the NW camera 110 via the PoE hub 130, and controls the NW camera 110 to capture images by sending control commands described below. The NW camera 110 is installed to capture images of the location where the package A to be delivered is loaded, and captures images of the package A under the control of the image processing device 100. The package A to be loaded is the result of stacking a plurality of packages with labels attached. The image processing device 100 also receives the images acquired by the NW camera 110 via the PoE hub 130. The image processing device 100 detects the images of the labels describing the barcodes from the received images, and reads the barcodes. The image processing device 100 compares the information of the barcodes read from the images with the information of the barcodes stored in the DB 120. Through this process, it is possible to check whether the packages, etc. have been delivered as planned. Although an example of delivering packages is described in this exemplary embodiment, this exemplary embodiment can be applied to the comparison when executing packages.

[0032] <Structure of the image processing device 100>

[0033] Figure 2 is a block diagram showing the hardware structure of the image processing device 100. The image processing device 100 includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, a random access memory (RAM) 203, a display 204, a hard disk drive (HDD) 205, an input device 206, a media drive 207, and an interface (I / F) 208. The CPU 201 reads the control program stored in the ROM 202 to execute various processes. The RAM 203 is used as the main memory of the CPU 201 and a temporary storage area such as a work area. The HDD 205 stores various data and various programs, etc. The display 204 displays various information. The input device 206 accepts various operations performed by the user. The media drive 207 reads data from a medium such as a Secure Digital (SD) card and writes data to the medium, for example. The I / F 208 communicates with external devices.

[0034] The functions and processes of the image processing device 100 described below are implemented by the CPU 201 reading the program stored in the ROM 202 or the HDD 205 and executing the program. Optionally, as another example, the CPU 201 may read a program stored in a storage medium such as an SD card instead of the ROM 202 or the HDD 205. As still another example, at least a part of the functions and processes of the image processing device 100 can be implemented by making a plurality of CPUs, a plurality of RAMs, a plurality of ROMs, and a plurality of storage devices cooperate with each other. Additionally, as still another example, at least a part of the functions and processes of the image processing device 100 can be implemented by using a hardware circuit.

[0035] <Structure of NW Camera 110>

[0036] Figure 3 is an external view of NW Camera 110. The pan drive unit 301 changes the orientation of the lens barrel unit 303 in the direction indicated by the pan direction 304 by driving the pan motor. The tilt drive unit 302 changes the orientation of the lens barrel unit 303 in the direction indicated by the tilt direction 305 by driving the tilt motor. In addition, the lens barrel unit 303 including the lens can be rotated in the direction indicated by the rotation direction 306 around the center position of the lens by being driven by the rotation motor. Further, the lens barrel unit 303 includes a focusing lens and a zoom lens, each of which is driven by a stepping motor. The whole of NW Camera 110 is covered by the dome 307.

[0037] Figure 4 is a block diagram showing the hardware structure of NW Camera 110. NW Camera 110 is an imaging device capable of communicating with an external device via a network. NW Camera includes a lens unit 401, a charge coupled device (CCD) unit 402, a signal processing unit 403, an image analysis unit 404, an encoding unit 405, and a communication processing unit 406. A description will be given of the processing until the image data captured by NW Camera 110 is delivered to the image processing device 100. The optical image obtained by the lens unit 401 is converted by the CCD unit 402 into red, green, and blue (RGB) digital data, and then sent to the signal processing unit 403. The signal processing unit 403 performs processing to convert the RGB digital data into digital data (image data) in the YCbCr 4:2:0 format or the YCbCr 4:2:2 format, processing to convert the size of the data into the required image size of the image to be sent, and various filtering processes. The processed image data is sent to both the image analysis unit 404 and the encoding unit 405 at the same time. Then, the image data is sent to the external device via the network by the communication processing unit 406.

[0038] The encoding unit 405 performs processing to encode and compress the image data into a predetermined format (for example, the H.264 format or the Joint Photographic Experts Group (JPEG) format). The communication processing unit 406 sends the H.264 video stream data or each JPEG still image data generated by the encoding unit 405 to the image processing device 100 according to network protocols such as the Transmission Control Protocol / Internet Protocol (TCP / IP), the Hypertext Transfer Protocol (HTTP), and the Real-Time Transport Protocol (RTP).

[0039] The image analysis unit 404 performs processing to analyze the captured image data and detect whether the subject or the image pattern of the specified condition is included in the target image. Each processing block of the signal processing unit 403, the image analysis unit 404, the encoding unit 405, and the communication processing unit 406 is connected to the CPU 411. The camera control unit 407 is connected to the motor drive unit 408 and the lens drive unit 410. According to the instruction from the CPU 411, the camera control unit 407 outputs control signals for the panning / tilting / rotation operation of the camera (movement in the panning direction, movement in the tilting direction, and rotation around the optical axis) and control signals for the zooming and autofocus (AF) operations.

[0040] In addition, the camera control unit 407 controls at least one of the visible range and the movable range of the NW camera 110 based on at least one of the visible range setting and the movable range setting stored in the RAM 413. The motor drive unit 408 includes a motor drive circuit and can change the imaging direction of the camera by driving the panning / tilting / rotation motor 409 in response to the control signal output from the camera control unit 407 and rotating the panning / tilting / rotation motor 409. The lens drive unit 410 includes a motor and a motor drive circuit for performing various drives such as AF and controls the lens drive unit 410 based on the control signal from the camera control unit 407.

[0041] The CPU 411 controls the operation of the entire device by executing the control program stored in the ROM 412. The CPU 411 is connected to the ROM 412, the RAM 413, and the flash (FLASH, registered trademark) memory 414. In addition, the CPU 411 is also connected to the signal processing unit 403, the image analysis unit 404, the encoding unit 405, and the communication processing unit 406, and controls each processing block by starting and stopping operations for each processing block, setting operation conditions, and obtaining operation results. The ROM 412 stores programs and data used by the CPU 411 for controlling the NW camera 110 (such as application processing, etc.).

[0042] The RAM 413 is a memory for writing and reading data when the CPU 411 executes the program stored in the ROM 412. The RAM 413 includes a working area and a temporary storage area used by the CPU 411 to execute the program for controlling the NW camera 110. The RAM 413 stores at least one of the visible range setting or the movable range setting. The visible range setting is used to specify the range of the field of view angle capable of capturing an image, and the movable range setting is used to specify the movable range in the panning direction, the tilting direction, and the zoom direction.

[0043] The CPU 411 changes the image shooting direction or the zoom magnification in response to a control command received from the image processing device 100 via the communication processing unit 406. When the CPU 411 receives a control command specifying the center position and the zoom magnification from the NW camera 110, it responds to the control command, controls the pan and tilt to set the specified position at the center of the image shooting, and controls the zoom to set the specified zoom magnification.

[0044] <Read>

[0045] Figure 5 It is a diagram showing a loaded package to be processed. In the present exemplary embodiment, as Figure 5 shown, the loaded package B with irregularly arranged labels is used as the processing object. The image processing device 100 uses the label model image of the overall image obtained by photographing the entire loaded package to perform a matching process to detect each area where the label model image appears, and determines the range for photographing the enlarged image based on the detection result.

[0046] <Preparation process>

[0047] Figure 6 It is a flowchart showing the preparation process performed by the image processing device 100.

[0048] In step S600, the CPU 201 adjusts the position (the position for photographing the overall image) of the image of the entire loaded package in response to the user's operation. While viewing the image of the loaded package displayed on the display 204, the user adjusts the pan, tilt, and zoom so that the entire loaded package falls within the image shooting range. The CPU 201 generates a control command according to the settings of the pan, tilt, and zoom adjusted based on the user's operation, and sends the control command to the NW camera 110. The NW camera 110 receives the control command from the image processing device 100, performs pan, tilt, and zoom based on the settings indicated by the control command to perform the image shooting process, acquires the overall image, and sends the overall image to the image processing device 100. The CPU 201 controls to display the received overall image on the display 204.

[0049] Next, in step S601, the CPU 201 specifies the position of the label area in the overall image. More specifically, while viewing the overall image displayed on the display 204, the user performs an operation to find the label area in the overall image and specifies the position. The CPU 201 responds to the user's operation and specifies the position of the label area in the overall image. The specified position of the label area is used as the position for photographing the enlarged image.

[0050] Next, in step S602, the CPU 201 creates a matching model (model image). More specifically, the CPU 201 extracts a label image from the overall image 700 shown in Figure 7 based on the position specified in step S601, and sets this label image as the matching model 701 of the label image.

[0051] Next, in step S603, the CPU 201 sets up the matching process. More specifically, the CPU 201 sets the object area for the matching process based on the frame 801 set by the user on the setting screen 800 shown in Figure 8 . The CPU 201 also sets the matching model in response to the user's operation. At this time, setting the matching model is to specify the matching model image that uses the matching model generated in step S602 as the reference image for the matching process performed in step S603. The set matching model is displayed in area 802. The CPU 201 also sets the matching parameters in response to the input to area 803. The CPU 201 also determines the execution order of the matching process. For example, the CPU 201 makes settings for performing the matching process in ascending / descending order of the X coordinate or Y coordinate.

[0052] When performing the matching process, the CPU 201 displays on the display 204 the Figure 9 test screen 900 shown. The overall image is displayed in area 901 of the test screen 900, and a frame 902 indicating the matching result of each area where the matching model appears is superimposed on this overall image. Figure 9 It indicates that all label areas 1 to 7 match the matching model, and the detection is successful. If the user refers to the matching result and finds any label area where the detection fails, the user can adjust the accuracy of the matching process by re-specifying the position of the captured magnified image in area 903 and / or resetting the zoom ratio on area 904. At least one of the label areas detected in the matching process is set as the position for the captured magnified image for testing.

[0053] Next, in step S604, if the execution test button 905 is pressed in the state where the position and zoom ratio of the captured magnified image are set on the test screen 900, the CPU 201 creates a control command based on this setting and sends this control command to the NW camera 110. The NW camera 110 receives the control command from the image processing device 100, performs panning, tilting, and zooming (PTZ) based on the settings indicated by the control command to perform image capture processing, acquires the magnified image, and sends the magnified image to the image processing device 100. Then, the CPU 201 displays the received magnified image in area 901. The user can check whether the barcode image is properly captured in the magnified image and adjust the zoom ratio.

[0054] Next, in step S605, the CPU 201 further performs read setting. The CPU 201 sets a rectangular area that is the object of reading the barcode, the type of the barcode, the number of barcodes, a dictionary, and the like.

[0055] Next, in step S606, the CPU 201 makes settings to store the information read in the reading process, which will be described below. More specifically, as Figure 10 shown, the CPU 201 creates a storage area for storing data.

[0056] <Overall Processing>

[0057] Figure 11 is a flowchart showing the overall processing according to the first exemplary embodiment.

[0058] In step S1100, the CPU 201 acquires a judgment threshold from the HDD 205. The judgment threshold is the value stored in step S1115, which will be described below, in the previous overall processing. Details of the judgment threshold will be described below. The judgment threshold is stored in the HDD 205. The HDD 205 corresponds to a storage unit.

[0059] Next, a description of over-detection will be given. In the present exemplary embodiment, as Figure 5 shown, the CPU 201 detects a region where the model image appears from the overall image of the loaded package with multiple labels attached, by using the matching process of the label model image. At this time, the undetected labels are not subjected to the reading process. For this reason, the number of undetected labels is reduced to zero. Thus, in the present exemplary embodiment, in the setting of the matching process in step S603 (in Figure 6 ), a low threshold is set for the matching parameter, which prevents undetected labels even if the detection becomes somewhat over-detection. In this way, the setting of detecting the number of regions that are more or less excessive than the actual number of labels is called "over-detection setting". In addition, the number of regions that do not include labels is called "the number of over-detection regions". Further, the number of regions that include labels but have not been detected is called "the number of undetected regions".

[0060] Figure 12A and Figure 12B are diagrams showing over-detection. Figure 12A shows a case where the number of over-detection regions is large but the number of undetected regions is zero. Although in Figure 12AAll parts 1 to 7 including the label have been detected in [[]], but parts 8 to 18 that do not include the label have been detected in area 1201. In this case, the number of over-detection areas is large (the number of objects for subsequent processing is large), thus increasing the overall processing time. However, since this system needs to make the number of undetected areas zero, this case is an appropriate case. On the other hand, Figure 12B shows a case where the number of over-detection areas is small but the number of undetected areas is not zero. In Figure 12B , parts 7 to 9 that do not include the label have been detected in area 1202, and the number of over-detection areas is less than that in the case of Figure 12A , but there are undetected labels in area 1203. In this system, the appropriate threshold of the matching parameter makes the number of over-detection areas as small as possible in the case shown in Figure 12A , rather than the case shown in Figure 12B .

[0061] Description returns to Figure 11 the flowchart in

[0062] In step S1101, the CPU 201 generates a control command under the conditions set in step S600 in ( Figure 6 ), and sends the control command to the NW camera 110. The NW camera 110 receives the control command from the image processing device 100, performs panning, tilting, and zooming based on the settings indicated by the control command for image shooting processing, acquires an image, and sends the image to the image processing device 100.

[0063] In step S1102, the CPU 201 performs a matching process on the overall image received in step S1101 based on the information set in step S603 in ( Figure 6 ), and detects the area where the label model image appears. In this exemplary embodiment, since the matching process is set to the above over-detection setting, an area with a detection quantity larger than the number of labels actually present in the overall image is detected. The area detected in step S1102 is hereinafter referred to as the detection area. In step S1102, the CPU 201 serves as a detection unit.

[0064] In step S1103, the CPU 201 performs the setting of the zoom ratio set in step S604 in ( Figure 6 ).

[0065] In step S1104, the CPU 201 sets the first detection area in the detection areas detected in step S1102 as the processing object.

[0066] In step S1105, the CPU 201 performs a detection determination process on the detection area set as the processing target. The detection determination process executed in step S1105 is hereinafter referred to as the first determination process.

[0067] <Preparation process for performing the first determination process>

[0068] First, a description of the preparation process for performing the first determination process will be given.

[0069] Figure 13 is a flowchart showing the preparation process for performing the first determination process. Figure 14A Shows the setting screen 1400 for setting the determination threshold. Figure 14B Shows the setting screen 1410 for setting the image processing area.

[0070] In Figure 13 In the flowchart shown, the CPU 201 makes settings to determine whether the detection area detected in Figure 11 step S1102 is a label area. In this exemplary embodiment, the CPU 201 makes settings to determine whether the detected area is a label area based on the presence or absence of features of the barcode image. For example, the CPU 201 designates the label area in the overall image as the image processing area and performs image processing on the image processing area. Then, the CPU 201 sets a threshold used as a determination criterion to determine the presence or absence of features of the barcode image for the average density and image feature amounts such as density standard deviation obtained through image processing. The threshold of the image feature amount used as the determination criterion is hereinafter referred to as the determination threshold. The determination threshold corresponds to the determination reference value.

[0071] In steps S1300 and S1301, the CPU 201 selects a reference image and sets the image processing area. More specifically, the CPU 201 selects the overall image (reference image) used as the reference for setting the determination threshold in the area 1402 of the setting screen 1400 shown in Figure 14A For the setting of the image processing area, the CPU 201 selects Figure 14BThe shape of the specified box 1401 in the area 1411 of the setting screen 1410 shown is such that the specified box 1401 with the specified shape is displayed on the reference image, and the image processing area is set based on the position and size of the specified box 1401. More specifically, the user uses the mouse as the input device 206 to specify the size of the specified box 1401 to match the label area, and operates the arrow icon in the operation area 1413 to specify the position of the specified box 1401. Alternatively, the user directly inputs the coordinate values of the starting point coordinates X and Y and the ending point coordinates X and Y in the area 1412, and specifies the position and size of the specified box 1401 to match the label area.

[0072] Subsequently, in step S1302, the CPU 201 sets the judgment threshold. Specifically, the CPU 201 sets the upper limit value (max) / lower limit value (mini) of the judgment threshold corresponding to the input of the area 1403 shown with respect to the average concentration and concentration deviation (standard deviation of concentration) of the image processing area set in the above step S1301. For example, when the assumed minimum average concentration of the barcode is 130, the upper limit value of the average concentration is set to 255, and the lower limit value of the average concentration is set to 130. In addition, when the assumed minimum concentration deviation of the barcode is 20, the upper limit value of the concentration deviation is set to 255, and the lower limit value of the concentration deviation is set to 20. In addition, the assumed maximum concentration that the barcode has at least and the assumed minimum concentration that the barcode has at most are also set in a similar manner. Figure 14A In the present exemplary embodiment, the average concentration and the concentration deviation are used as the judgment threshold. On the other hand, as long as the feature quantity related to the color of the image is image information that can be obtained by performing image processing, the feature quantity related to the color of the image can be used together with the feature quantities related to the concentration of the image (such as the average concentration and the concentration deviation, etc.) or instead of the feature quantities related to the concentration of the image.

[0073] The description returns to

[0074] the flowchart in Figure 11 .

[0075] In step S1105, the CPU 201 performs a first judgment process on the detection area set as the processing object. More specifically, the CPU 201 judges whether each of the above average concentration and concentration deviation of the image processing area set in step S1301 for the detection area in the overall image is within ( Figure 13 of) Figure 13within the range of the determination threshold set in step S1302. For a detection area determined to be within the range of the determination threshold, the CPU 201 performs over-detection determination processing for the magnified image (in step S1110) on the magnified image obtained by photographing a magnified image of the detection area on the detection area, and determines whether the magnified image is a barcode image. Thus, even if in step S1105 the CPU 201 determines that both the average concentration and the concentration deviation are within the range of the determination threshold, this area is not fixed as the area including the barcode. In other words, determining in step S1105 that both the average concentration and the concentration deviation are within the range defined by the determination threshold means determining that the detection area is a candidate for the area including the barcode (i.e., not an over-detection area). On the other hand, determining that both the average concentration and the concentration deviation are outside the range defined by the determination threshold means determining that the detection area is not the area including the barcode (i.e., is an over-detection area). The CPU 201 causes the RAM 203 to hold the average concentration and the concentration deviation of the detection area for the first determination process. Each time the first determination process is executed, the average concentration and the concentration deviation held in the RAM 203 are updated.

[0076] In step S1106, the CPU 201 determines whether the detection area set as the processing object is an over-detection area. When the CPU 201 determines that the detection area is an over-detection area (i.e., not the area including the barcode) (in step S1106, it is "yes"), the process proceeds to step S1116. The detection area determined to be an over-detection area is excluded from the object for photographing the magnified image and the object for performing the reading process. This can reduce the overall processing time. On the other hand, when the CPU 201 determines that the detection area is not an over-detection area (i.e., is a candidate for the area including the barcode) (in step S1106, it is "no"), the process proceeds to step S1107. In this way, the CPU 201 serves as the first determination unit in steps S1105 and S1106.

[0077] Subsequently, in step S1107, the CPU 201 sets the position of the detection area set as the processing object as the center position for photographing the magnified image.

[0078] In step S1108, the CPU 201 generates a control command for causing the NW camera 110 to adjust panning, tilting, and zooming based on the zoom ratio set in step S1103 and the center position of the magnified captured image set in step S1107, and sends the control command to the NW camera 110. The NW camera 110 receives the control command from the image processing device 100 and sets panning, tilting, and zooming based on the settings indicated by the control command. Additionally, the zoom ratio is set on the first detection area but not on the second and subsequent detection areas. Thereby, the NW camera 110 adjusts panning and tilting for the second or subsequent detection areas. In steps S1107 and S1108, the CPU 201 functions as an imaging control unit.

[0079] In step S1109, the NW camera 110 performs image capture processing after panning, tilting, and zooming in accordance with the settings made in step S1108, acquires a magnified image, and sends the magnified image to the image processing device 100. The CPU 201 receives the magnified image from the NW camera 110. In step S1109, the CPU 201 functions as a receiving unit.

[0080] In step S1110, the CPU 201 performs an over-detection determination process on the magnified image received in step S1109. The over-detection determination process performed in step S1110 is hereinafter referred to as the second determination process. More specifically, the CPU 201 first executes preparatory processing to perform a second determination process similar to the processing shown in the flowchart described with reference to Figure 13 and makes various settings. Figure 15A Displays a setting screen 1500 similar to the setting screen 1400 shown in Figure 14A to set a threshold for determination. Figure 15B Displays a setting screen 1510 similar to the screen shown in Figure 14B to set an image processing area. The CPU 201 determines whether the magnified image is a barcode image (i.e., not an over-detection area) or not a barcode image (i.e., an over-detection area) based on various set values set in the preparatory processing. For example, similar to the first determination process, the CPU 201 determines whether the magnified image is a barcode image based on the threshold of the image feature amount of the set image processing area.

[0081] In the second determination process, since the image of the object to be judged is a magnified image, the resolution of the image of the photographed object is high. Thus, character recognition processing or shape matching processing, etc. can be used as a means for judging over-detection. For example, in Figure 15A , the CPU 201 performs an instruction for a predetermined character in area 1502 (in Figure 15AThe setting of the model image (e.g., "ABC" in the example) and the setting of the matching parameters are based on the input to region 1503. In this case, as the second determination process, the CPU 201 performs a matching process on the image processing region set by the specified frame 1501 in the enlarged image according to the set information, and determines whether a predetermined character appears.

[0082] In step S1111, the CPU 201 determines whether the enlarged image received in step S1109 is an image of an over-detection region. When the CPU 201 determines that the enlarged image is an image of an over-detection region (i.e., "Yes" in step S1111), the process proceeds to step S1114. On the other hand, when the CPU 201 determines that the enlarged image is not an image of an over-detection region (i.e., "No" in step S1111), the process proceeds to step S1112. In steps S1110 and S1111, the CPU 201 serves as the second determination unit.

[0083] As a result of the determination process, when the CPU 201 determines that the enlarged image is an image of an over-detection region (i.e., not a barcode image) (i.e., "Yes" in step S1111), the process proceeds to step S1114.

[0084] In step S1114, the CPU 201 identifies the region in the overall image captured in step S1101 that corresponds to the enlarged image received in step S1109. The identified region at this time is the detection region that has undergone the first determination process. In addition, the CPU 201 obtains the average concentration and the concentration deviation for the first determination process performed on the identified region. These values are in a state of being held in the RAM 203 as the latest values output in the process of step S1105, and thus can be easily obtained.

[0085] Subsequently, in step S1115, the CPU 201 updates the judgment threshold used in the first judgment process based on the average concentration and the concentration deviation obtained in step S1114, and stores the judgment threshold in the HDD 205. More specifically, in this execution flow, as described above, the initial value of the judgment threshold is set as follows: the upper limit value of the average concentration is set to 255, the lower limit value of the average concentration is set to 130, the upper limit value of the concentration deviation is set to 255, and the lower limit value of the concentration deviation is set to 20. In this set state, assuming that the average concentration is 128 and the concentration deviation is 19 as the image feature amounts obtained in step S1114. In this case, the CPU 201 updates the lower limit value of the average concentration to 128 and updates the lower limit value of the concentration deviation to 19. Additionally, in the subsequent process, assuming that the average concentration is 129 and the concentration deviation is 18 as the image feature amounts obtained in step S1114. In this case, in step S1115, the CPU 201 keeps the average concentration at the currently stored 128 (does not update the average concentration with the current value). On the other hand, the CPU 201 updates the currently stored concentration deviation of 19 to 18. In this way, in the first judgment process, the allowable range of the judgment threshold of the first judgment process is changed so that more detection areas are judged as over-detection areas (i.e., areas that do not include barcodes).

[0086] However, if the allowable range of the judgment threshold for judging the detection area as an over-detection area is too wide in step S1105, there is a possibility of an undetected area as described above. To avoid such a situation, the CPU 201 can also set change limit values for the upper limit value / lower limit value of the judgment threshold of the first judgment process to prevent the threshold from becoming a certain value or greater (or smaller). More specifically, if both the average concentration and the concentration deviation obtained in step S1114 are within the range defined by the change limit values set as the upper limit value / lower limit value of the judgment threshold, the CPU 201 updates the judgment threshold. If both the average concentration and the concentration deviation are outside the range defined by the change limit values set as the upper limit value / lower limit value of the judgment threshold, the CPU 201 does not update the judgment threshold. After completing the process in step S1115, the process proceeds to step S1116. In steps S1114 and S1115, the CPU 201 serves as an update control unit.

[0087] In step S1116, the CPU 201 determines whether the detection area that has undergone the first determination process is the last detection area in accordance with the detection order in step S1102. When the CPU 201 determines that the detection area is the last detection area (Yes in step S1116), the process proceeds to step S1118. When the CPU 201 determines that the detection area is not the last detection area (No in step S1116), the process proceeds to step S1117.

[0088] In step S1117, the CPU 201 sets the next detection area as the processing object. Thereafter, the process returns to step S1105. In this way, the first determination process is sequentially executed for a plurality of detection areas. Although the center position of the captured magnified image is changed in step S1107 and panning and tilting are readjusted in step S1108 to capture an image, the NW camera 110 does not perform an operation of reducing once and then magnifying again. This can reduce the processing time required for the zoom drive of the NW camera 110.

[0089] In step S1118, the CPU 201 displays the results of the first determination process and the second determination process on the display 204 in association with the respective positions of the detection areas detected in step S1102 in the overall image captured in step S1101. A detailed description will be given below with reference to Figure 18A will be given in detail. In addition, the CPU 201 displays the display item whose threshold value was updated in step S1115 on the display 204. A detailed description will be given below with reference to Figure 18B will be given in detail. In addition, the notification method is only a method of notifying that the threshold value has been updated, and a method other than the display method can be used to make the notification. In step S1118, the CPU 201 functions as a display control unit and a notification unit. When the processing in step S1118 is completed, the sequence of the above-described overall processing procedure ends.

[0090] Next, a description will be given of the case where, as a result of the determination process in step S1111, the magnified image is not an image of an over-detection area (i.e., is a barcode image).

[0091] In step S1112, the CPU 201 performs a code reading process on the magnified image received from the NW camera 110 in step S1109 based on the information set in step S605 in ( Figure 6 ). Subsequently, in step S1113, the CPU 201 uses, as the one based on the one in ( Figure 6The read information obtained as a result of the code reading process performed on the information set in step S606 in ) is stored in a storage unit such as HDD 205. After the processing of step S1113 is completed, the processing proceeds to step S1116. In steps S1112 and S1113, the CPU 201 functions as a reading unit.

[0092] In addition, in the case where the code reading process in step S1112 fails due to the accuracy of the pan and tilt control of the NW camera 110 in the pan and tilt settings performed in step S1108, the CPU 201 can set the zoom magnification, perform the matching process, and gradually set the pan and tilt. More specifically, the CPU 201 sets the zoom magnification in step S1103 to be lower than the zoom magnification setting that enables the code reading process in step S1112, and after shooting the magnified image in step S1109, uses the label model image again for the matching process.

[0093] As a result of performing the matching process again, the CPU 201 sets the center position of the detection area of the coordinates close to the center of the screen, sets the pan and tilt, shoots the magnified image, and performs the code reading process.

[0094] Figure 16 An example of the captured image screen displayed on the display 204 when shooting the magnified image in (step S1109) is shown. The overall image captured in step S1101 is displayed in area 1601, and the frame 1602 indicating the result of the matching process in step S1102 is overlapped on the overall image. In addition, the magnified image is displayed in area 1603. Figure 16 An example of the magnified image showing the fifth detection area in the detection order according to step S1102 is shown.

[0095] Figure 17 An example of the read image screen 1700 displayed on the display 204 when performing the code reading process in (step S1112) is shown. The results 1711, 1712, and 1713 of the code reading process are displayed on the read image screen 1700.

[0096] Figure 18A An example of the display screen showing the following on the overall image: the detection area determined to be an over-detection area (i.e., not the area including the barcode) in the first determination process; the detection area corresponding to the magnified image determined to be an over-detection area (i.e., the image is not a barcode image) in the second determination process; the detection area corresponding to the magnified image where the code reading process is normally performed; and the detection area corresponding to the magnified image where the code reading process fails.

[0097] Specifically, a rectangular display box is displayed in the detected detection area in step S1102, and the detection order in step S1102 is displayed as a number within the display box. Additionally, as the judgment result in the detection area determined to be an over-detection area (i.e., not an area including a barcode) in the first judgment process, a solid circle (●) is displayed. As the judgment result in the detection area corresponding to the magnified image and determined to be an over-detection area (i.e., the image is not a barcode image) in the second judgment process, a solid triangle (▲) is displayed. As the judgment result in the detection area corresponding to the magnified image where the code reading process is normally performed, a circle (○) is displayed. As the judgment result in the detection area corresponding to the magnified image where the code reading process fails, a cross (x) is displayed. Additionally, the CPU 201 can perform such display on the display 204 in real time, or can store the display image as image data in the HDD 205. Although the description has been given under the assumption that the CPU 201 displays the judgment results and reading results in each detection area, the CPU 201 can log the judgment results and reading results for each detection area. For example, the CPU 201 can associate the results with the detection order in step S1102 by classifying the circle (○) as 1 (normal reading), the cross (x) as 99 (abnormal reading), the solid circle (●) as 11 (the result of the first over-detection judgment process is over-detection), and the solid triangle (▲) as 21 (the result of the second over-detection judgment is over-detection), and store the results as log data in the HDD 205.

[0098] Figure 18B An example of a display screen indicating the update of the threshold in step S1105 is shown. More specifically, a display item indicating that the threshold is updated is displayed in the upper right part of the display 204. The position where the display item is displayed can be set by the user, or can be set by the program in a unique area. Additionally, the display item can be a freely selectable mark. Also, the CPU 201 can store the following log data in the HDD 205: as the result of the overall process, the log data is 1 when the threshold is updated, and the log data is 0 when the threshold is not updated. Additionally, when the process using the threshold ends or when the overall process ends, the CPU 201 can hide the display item simultaneously.

[0099] As described above, in the image processing system according to the first exemplary embodiment, the CPU 201 over-detects the candidates of the area including the barcode in the overall image so as not to miss the undetected area, and excludes the area not including the barcode from the subsequent processing steps based on the judgment result in the first judgment processing. In addition, the CPU 201 can update the judgment threshold used in the first judgment processing based on the judgment result in the second judgment processing using the enlarged image. With this processing, the judgment threshold used in the first judgment processing can be appropriately set, and the detection area having features similar to the features of the detection area corresponding to the enlarged image judged as the over-detected area in the second judgment processing can be excluded from the subsequent processing steps in the next and subsequent overall processing. Therefore, the structure can reduce the overall processing time while preventing the undetected area from occurring.

[0100] Next, refer to Figure 19 An image processing system according to a second exemplary embodiment is described.

[0101] In the first exemplary embodiment described above, the first determination process is executed based on the determination threshold value acquired at the start of the overall process. Figure 11 , the judgment threshold value will not be reflected in the first judgment process. Therefore, when a detection area having features similar to those of the detection area corresponding to the enlarged image judged to be an over-detected area in the second judgment process appears while the overall process is being performed, the detection area that appears again will not be excluded from the subsequent processing steps. Thus, in the second exemplary embodiment, the CPU 201 acquires the judgment threshold value after performing the second judgment process on the previous detection area and before performing the first judgment process on the next detection area. The hardware structure of the image processing system according to the second exemplary embodiment is similar to that of the first exemplary embodiment.

[0102] Figure 19 is a flowchart illustrating overall processing according to the second exemplary embodiment. Figure 19 The flowchart shown is similar to Figure 11 The illustrated flowchart is different in that the processing in step S1904 corresponding to step S1100 is performed between steps S1903 and S1905 corresponding to steps S1104 and S1105 , respectively.

[0103] As described above, in the image processing system according to the second exemplary embodiment, while performing the overall processing, the updated determination threshold is reflected in the first determination processing. Thus, when a detection region having features similar to those of the detection region corresponding to the magnified image of the over-detection region determined in the second determination processing appears again while performing the overall processing, the reappearing detection region can be quickly excluded from the subsequent processing steps. This can reduce the overall processing time.

[0104] Although the present disclosure has been described in detail with reference to the exemplary embodiments, the present disclosure is not limited to the specific exemplary embodiments and includes various embodiments without departing from the gist of the present disclosure. In addition, each of the above exemplary embodiments merely indicates one exemplary embodiment of the present disclosure, and they can be combined appropriately.

[0105] For example, in each of the above exemplary embodiments, when the detection region is determined as an over-detection region (not a bar code image) in the second determination processing, the determination threshold used in the first determination processing is updated. Instead of the structure in which the determination threshold used in the first determination processing is updated according to the determination result in the second determination processing in this way, a structure in which the determination threshold used in the first determination processing is updated according to the determination result in the code reading processing can be adopted. More specifically, when the result of the code reading processing indicates failure, the CPU 201 updates the determination threshold used in the first determination processing. In this case, in the first exemplary embodiment, the processing in steps S1100 and S1111 becomes unnecessary, and when it is determined that the code reading processing fails after the processing in step S1112, the processing proceeds to step S1114.

[0106] Aspects of the embodiments can reduce the overall processing time while preventing omission of detection of an image capture object.

[0107] Other embodiments

[0108] Embodiments of the present disclosure can also be implemented by a method in which software (program) that executes the functions of the above-described embodiments is provided to a system or apparatus via a network or various storage media, and a computer or a central processing unit (CPU) or a micro processing unit (MPU) of the system or apparatus reads and executes the program.

[0109] Although the present disclosure has been described with reference to the exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications as well as equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: a detection unit configured to detect a plurality of regions from an image; a first determination unit configured to perform a first determination process on each of the plurality of regions to determine whether each of the plurality of regions is a candidate for a region including a specific object based on a determination reference value stored in a storage unit; a control unit configured to control an imaging device to capture a magnified image of a region that is a candidate for a region including the specific object among the plurality of regions in the first determination process; a reception unit configured to receive the captured magnified image; a second determination unit configured to perform a second determination process on the magnified image to determine whether the magnified image includes the specific object; and an update control unit configured to identify a region corresponding to a magnified image determined not to include the specific object in the second determination process, and update the stored determination reference value based on image information of the identified region, wherein, in the second determination process, a reading process of the magnified image is performed, and it is determined that the magnified image as a reading object includes the specific object when the reading process is successful, and it is determined that the magnified image as a reading object does not include the specific object when the reading process fails.

2. The image processing apparatus according to claim 1, Among them, in the first determination process, image information for each of the plurality of regions is used to determine whether each of the plurality of regions is a candidate for a region including the specific object, and wherein, the determination reference value is updated based on the image information used in the first determination process.

3. The image processing apparatus according to claim 1, Among them, the image information indicates an image feature amount of the region.

4. The image processing apparatus according to claim 3, Among them, the determination reference value is a predetermined threshold value of the image feature amount, and wherein, when the image feature amount of the region is within a range defined by the predetermined threshold value, it is determined that the region is a candidate for a region including the specific object.

5. The image processing apparatus according to claim 4, Among them, a limit value is set for the determination reference value, and wherein, when the image feature amount of the region does not exceed the limit value, the determination reference value is updated, and when the image feature amount of the region exceeds the limit value, the determination reference value is not updated.

6. The image processing apparatus according to claim 3, Among them, the image feature amount is a feature amount related to the density of the image and / or a feature amount related to the color of the image.

7. The image processing apparatus according to claim 1, Among them, a plurality of determination reference values are stored in the storage unit, and wherein, a part of the plurality of determination reference values is updated.

8. The image processing apparatus according to claim 1, Among them, the specific object is at least one of a bar code, a quick response code (QR code), and a character string.

9. The image processing apparatus according to claim 1, further comprising: A reading unit configured to perform a reading process on a magnified image determined to include the specific object in the second determination process.

10. The image processing apparatus according to claim 1, Among them, In the second determination process, based on the image information of the magnified image, it is determined whether the magnified image includes the specific object.

11. The image processing apparatus according to claim 1, Among them, In the second determination process, based on the result obtained by performing character recognition processing and / or matching processing on the magnified image, it is determined whether the magnified image includes the specific object.

12. The image processing apparatus according to claim 1, Among them, The first determination process and the second determination process are sequentially performed on the plurality of regions, and after performing the second determination process on the previous region and before performing the first determination process on the next region, the determination reference value is obtained from the storage unit.

13. The image processing apparatus according to claim 1, Among them, The first determination process is sequentially performed on the plurality of regions, and before performing the first determination process on the first region, the first determination process is performed on the last region based on the determination reference value obtained from the storage unit.

14. The image processing apparatus according to claim 1, Among them, A matching process is performed between the image and a model image including the specific object, and the region is detected from the image based on the result of the matching process.

15. The image processing apparatus according to claim 1, Among them, The determination result in the first determination process and the determination result in the second determination process are displayed in association with the positions of the plurality of regions in the image.

16. The image processing apparatus according to claim 1, Among them, In the case where the determination reference value is updated, a notification indicating that the determination reference value is updated is made.

17. A control method for an image processing apparatus, the control method comprising: Detecting a plurality of regions included in the overall image from the overall image; Performing a first determination process on each of the plurality of regions in the overall image to determine whether each of the plurality of regions is a candidate for a region including an object to be processed based on a determination reference value stored in a storage unit; Controlling a imaging device to capture a magnified image of a region that is a candidate for a region determined to include the object in the first determination process among the plurality of regions in the overall image; Receiving the captured magnified image from the imaging device; Performing a second determination process on the magnified image to determine whether the magnified image is an image of the object; And Identifying a region in the overall image corresponding to the magnified image determined not to be an image of the object in the second determination process, and controlling to update the determination reference value based on the image information of the identified region. Among them, in the second determination process, a reading process of the enlarged image is performed, so as to determine that the enlarged image as the reading object includes the object when the reading process is successful, and determine that the enlarged image as the reading object does not include the object when the reading process fails.

18. A non-transitory computer-readable storage medium storing a program for a computer to execute a method, the method comprising: detecting a plurality of regions included in the overall image from the overall image; performing a first determination process on each of the plurality of regions in the overall image to determine whether each of the plurality of regions is a candidate for a region including an object to be processed based on a determination reference value stored in a storage unit; controlling an imaging device to capture an enlarged image of a region that is a candidate for a region including the object determined in the first determination process among the plurality of regions in the overall image; receiving the captured enlarged image from the imaging device; performing a second determination process on the enlarged image to determine whether the enlarged image is an image of the object; and identifying a region in the overall image corresponding to the enlarged image determined not to be an image of the object in the second determination process, and performing control to update the determination reference value based on the image information of the identified region; Among them, in the second determination process, a reading process of the enlarged image is performed, so as to determine that the enlarged image as the reading object includes the object when the reading process is successful, and determine that the enlarged image as the reading object does not include the object when the reading process fails.

Images