Image Processing Device

Abstract

To provide an image processing apparatus capable of creating an extraction image by satisfactorily extracting an imaging-target object from each frame of a video even when the imaging-target object is a movable object such as a vehicle or the like.SOLUTION: In an image processing system 1, a server which functions as an image processing apparatus includes an arithmetic processing unit which selects at least one frame from a storage unit storing a plurality of frames captured by an imaging apparatus, and outputs an extraction image extracted from the selected frame. When an update trimming range allowing for an evaluation value higher than an evaluation value of an image of an initial trimming range set in the selected frame can be set, an image trimmed in the update trimming range is output as an extraction image. The evaluation value of the image of the trimming range is set based on at least one evaluation item of the following evaluation items: a positional relationship of a target object in the image of the trimming range, a size of the target object, a vividness of the target object, a brightness of the image of the trimming range, and a degree of cutoff of the target object in the image of the trimming range.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an image processing device. [Background technology]

[0002] There have been known imaging devices that capture images of a subject, acquire image data, and detect a specific object from the acquired image data. For example, in an image processing device described in JP 2009-94946 A, one or more detected objects (K1 to K4) are judged as valid or invalid based on information including one or more of the object's size, position, amount of change, and focus evaluation value. Then, portrait right protection processing is performed on the object (e.g., K4) that is judged to be an invalid object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-94946 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when posting (uploading) to a social networking service (hereinafter, referred to as "SNS"), there is a demand to be able to trim and extract good images from a video and post them on the SNS or the like.

[0005] However, when the captured object is a moving object such as a car, it is difficult to properly extract the captured object from each frame of the video and create an extracted image.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can create an extracted image in which an object to be captured is well extracted from each frame of a video, even if the object to be captured is a moving object such as a car. [Means for solving the problem]

[0007] The image processing device includes a memory unit in which a plurality of frames captured by an imaging device that captures moving images are stored, and a processing unit that selects at least one frame from the plurality of frames stored in the memory unit and outputs an extracted image extracted from the selected frame, wherein the processing unit sets an initial cropping range to the selected frame, and if an updated cropping range can be set that is higher than the evaluation value of the image in the initial cropping range, outputs the image cropped with the updated cropping range as the extracted image, and if a cropping range higher than the evaluation value of the image in the initial cropping range cannot be set, outputs the image within the initial cropping range as the extracted image, and the evaluation value of the image in the cropping range is set based on at least one evaluation item of the positional relationship of objects in the image in the cropping range, the size of the object in the image in the cropping range, the sharpness of the object in the image in the cropping range, the brightness of the image in the cropping range, and the degree of cut-off of the object in the image in the cropping range.

[0008] The processing unit selects a first selected frame and a second selected frame from the multiple frames, and outputs the first extracted image if the evaluation value of the first extracted image extracted from the first selected frame is higher than the evaluation value of the second extracted image extracted from the second selected frame.

[0009] The processing unit lowers the evaluation value when a pre-registered excluded object is captured within the image of the trimming range. The object is a vehicle.

[0010] The processing unit sets an initial trimming range having a different composition based on the position of a vehicle as an object.

[0011] The processing unit acquires in advance at least one of license plate information of a target vehicle as an object and appearance information that specifies the appearance of the target vehicle, and the processing unit selects a frame in which the target vehicle is imaged from a plurality of frames, and crops the selected frame to include the target vehicle, thereby creating an extracted image.

[0012] The image processing device further includes a memory in which an extraction processing model is stored, and the extraction processing model is a trained model that receives an image including an object to be imaged as input and outputs an extracted image having an evaluation value equal to or greater than a predetermined value.

[0013] The pre-learning model of the trained model (extraction processing model) used in the extraction processing includes, for example, a neural network. The neural network is a well-known neural network used in image recognition processing by deep learning. Examples of such neural networks include a convolutional neural network (CNN) and a recurrent neural network (RNN). Effect of the Invention

[0014] According to the image processing device according to the present disclosure, even if the captured object is a moving object such as a car, it is possible to create an extracted image in which the captured object is suitably extracted from each frame of a video. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an image processing system 1. [Diagram 2] FIG. 2 is a perspective view showing the photographing system 2. [Diagram 3] 1 is a diagram showing one frame FR0 of video data captured by the identification camera 10. FIG. [Figure 4] 1 is a diagram showing one frame FR1 of video data captured by viewing camera 11. FIG. [Diagram 5]1 is a block diagram showing a configuration of an image processing system 1. FIG. [Figure 6] FIG. 13 is a diagram showing a schematic diagram of a cut-out video CV0. [Figure 7] 1 shows an example of a frame on which object detection processing has been performed. [Figure 8] FIG. 11 is a flow diagram showing the flow from when a specific video CV2 is generated to when a highly rated image is selected. [Figure 9] FIG. 11 is a flow diagram showing the extraction process in S200. [Figure 10] FIG. 1 shows a frame FR in which an initial trimming range TR0(1) is set. [Figure 11] FIG. 13 shows a frame FR in which an initial trimming range TR0(2) is set. [Figure 12] FIG. 13 shows a selected frame SF in which a trimming range TR1 has been set. [Figure 13] FIG. 13 is a diagram for explaining an example of a trained model (extraction processing model) used in the extraction processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <System configuration> FIG. 1 is a diagram illustrating an image processing system 1. As shown in FIG.

[0017] The image processing system 1 includes an image capturing system 2, a server 3, and an output terminal 4. The output terminal 4 is, for example, a user's smartphone or personal computer. The server 3 functions as an image processing device.

[0018] The photography system 2 and the server 3 are capable of communicating with each other via a network 5 , and the server 3 and the output terminal 4 are also capable of communicating with each other via the network 5 .

[0019] Fig. 2 is a perspective view showing the photographing system 2. The photographing system 2 includes an identification camera 10, a viewing camera 11, a control device 12, and a communication unit 13. The identification camera 10 is a camera that photographs vehicles traveling on a road. In the example shown in Fig. 2, the imaging direction of the identification camera 10 is different from the imaging direction of the viewing camera 11. The identification camera 10 transmits captured video data to the control device 12, and the viewing camera 11 transmits captured video data to the control device 12. The video includes multiple frames.

[0020] 3 is a diagram showing one frame FR0 of video data captured by the identification camera 10. In this frame FR0, a road 60 and a plurality of vehicles 9A, 9B traveling on the road 60 are captured.

[0021] 4 is a diagram showing one frame FR1 of video data captured by viewing camera 11. Frame FR1 includes images of a road 61, a mountain 62, and a tree 63. Note that a capturing area R1 indicates the area captured by viewing camera 11.

[0022] 5 is a block diagram showing the configuration of the image processing system 1. The output terminal 4 includes an input unit 30 and a communication unit 31. The input unit 30 is, for example, a screen having a touch panel function.

[0023] The photographing system 2 includes an identification camera 10, an observation camera 11, a control device 12, and a communication unit 13.

[0024] The control device 12 includes a vehicle extraction unit 14 , a license plate recognition unit 15 , a matching processing unit 16 , a target vehicle selection unit 17 , a feature extraction unit 18 , a storage unit 19 , and a video clipping unit 20 .

[0025] The server 3 includes a communication unit 30, a storage unit 31, and a calculation processing unit 32. The calculation processing unit 32 includes an object detection unit 34, a target vehicle identification unit 35, a frame extraction unit 36, an extraction processing unit 49 including an image processing unit 37 and an evaluation unit 38, an album creation unit 39, a web service management unit 40, and an image capture system management unit 41. The storage unit 30 includes an image storage unit 301 and a registration information storage unit 302.

[0026] In the image processing system 1 configured as above, a user inputs vehicle information relating to a vehicle to be imaged using the output terminal 4. An example of the vehicle to be imaged is a vehicle used by the user.

[0027] The vehicle information includes, for example, information indicating the vehicle number (license plate number) of the image-captured vehicle, the vehicle model, and the color. The output terminal 4 transmits the terminal ID of the output terminal 4 and the vehicle information VI to the server 3. The server 3 stores the vehicle information VI and the terminal ID in the storage unit 31.

[0028] In the photographing system 2, the identification camera 10 outputs the identification video to the vehicle extraction unit 14. The vehicle extraction unit 14 performs a vehicle extraction process to extract vehicles from the identification video. The vehicle extraction unit 14 extracts vehicles in general, not just the target vehicle, from the identification video. This process is also referred to as a "vehicle extraction process." For the vehicle extraction process, for example, a trained model generated by machine learning technology such as deep learning can be used. In this example, the vehicle extraction unit 14 is realized by a "vehicle extraction model."

[0029] The vehicle extraction unit 14 outputs a video from which a vehicle has been extracted (a frame including a vehicle) from the identification video to the number recognition unit 15 and also to the matching processing unit 16.

[0030] The number recognition unit 15 recognizes the number of the license plate from the video in which the vehicle is extracted by the vehicle extraction unit 14. This process is also referred to as "number recognition process." A trained model generated by machine learning techniques such as deep learning can also be used for the number recognition process. In this example, the number recognition unit 15 is realized by a "number recognition model." The number recognition unit 15 outputs the recognized number to the matching processing unit 16. The number recognition unit 15 also outputs the recognized number to the communication unit 13. As a result, the number of each vehicle is transmitted to the server 3.

[0031] The matching processing unit 16 associates the vehicle extracted by the vehicle extraction unit 14 with the number recognized by the number recognition unit 15. This processing is also referred to as a "matching processing". Specifically, referring again to FIG. 3, a situation in which two vehicles 9A, 9B are extracted and two numbers 81, 82 are recognized will be described as an example. The matching processing unit 16 calculates the distance between the number and the vehicle for each number (the distance between the coordinates of the number on the frame and the coordinates of the vehicle). Then, the matching processing unit 16 matches the number with a vehicle with a short distance between the number. In this example, the distance between the number 81 and the vehicle 9A is shorter than the distance between the number 81 and the vehicle 9B, so the matching processing unit 16 associates the number 81 with the vehicle 9A. Similarly, the matching processing unit 16 associates the number 82 with the vehicle 9B. The matching processing unit 16 outputs the result of the matching processing (the vehicle with the associated number) to the target vehicle selection unit 17.

[0032] The target vehicle selection unit 17 selects, as a target vehicle, a vehicle whose number matches the number of the target vehicle (received from the server 3) from among the vehicles whose numbers have been associated by the matching process. The target vehicle selection unit 17 outputs the vehicle selected as the target vehicle to the feature extraction unit 18.

[0033] The feature extraction unit 18 extracts the feature of the target vehicle by analyzing a video including the target vehicle. More specifically, the feature extraction unit 18 calculates the running speed of the target vehicle based on the temporal change of the target vehicle in the frames including the target vehicle (for example, the movement amount of the target vehicle between frames, the change amount of the size of the target vehicle between frames). In addition to the running speed of the target vehicle, the feature extraction unit 18 may calculate, for example, the acceleration (deceleration) of the target vehicle. Furthermore, the feature extraction unit 18 extracts information on the appearance (body shape, body color, etc.) of the target vehicle using a known image defect recognition technology. The feature extraction unit 18 outputs the feature (running state and appearance) of the target vehicle to the video cutout unit. Furthermore, the feature extraction unit 18 outputs the feature of the target vehicle to the communication unit 13. As a result, the feature of the target vehicle is transmitted to the server 3.

[0034] The storage unit 19 temporarily stores the viewing video. The storage unit 19 is typically a ring buffer (circular buffer) and has a circular storage area in which the beginning and end of a one-dimensional array are logically linked. A newly captured viewing video is stored in the storage unit 19 for a predetermined time that can be stored in the storage area. Viewing videos (old videos) that exceed the predetermined time are automatically deleted from the storage unit 19.

[0035] The video clipping unit 20 clips out a portion of the viewing video stored in the storage unit 19 that is likely to include the target vehicle, based on the feature values ​​(traveling speed, acceleration, body shape, body color, etc., of the target vehicle) extracted by the feature value extraction unit 18. To explain in more detail, the distance between the point photographed by the identification camera 10 and the point photographed by the viewing camera 11 is known. Therefore, if the traveling speed (and acceleration) of the target vehicle is known, the video clipping unit 20 can calculate the time difference between the timing at which the target vehicle is photographed by the identification camera 10 and the timing at which the target vehicle is photographed by the viewing camera 11. The video clipping unit 20 calculates the timing at which the target vehicle is photographed by the viewing camera 11, based on the timing at which the target vehicle is photographed by the identification camera 10 and the above-mentioned time difference.

[0036] Then, the video clipping unit 20 clips out the video from the viewing video so as to include a video from when the target vehicle starts to enter the photographing area R1 until when it moves out of the photographing area R1. In this manner, the video clipping unit 20 generates a clipped video CV0.

[0037] 6 is a diagram showing a schematic diagram of an extracted video CV0 The extracted video CV0 includes a plurality of frames FRS0 to FRE0.

[0038] The frame FRS0 is a frame in a state where the vehicle 9A, which is a target vehicle, starts to enter the photographing area R1, and the frame FRE0 is a frame in a state where the vehicle 9A is about to exit the photographing area R1.

[0039] The video clipping unit 20 transmits the clipped video CV0 to the communication unit 13. The communication unit 13 transmits the clipped video CV0 to the server 3.

[0040] In the server 3, the storage unit 31 stores in the image storage unit 301 the cut-out video CV0, extracted images (described later), information such as highly rated images obtained by arithmetic processing by the server 3, and various types of evaluation information.

[0041] The registration information storage unit 302 stores registration information related to the vehicle photography service. The registration information includes personal information of a user who has applied for the vehicle photography service and vehicle information of the user. The personal information of the user includes information related to the user's identification number (ID), name, date of birth, address, telephone number, email address, etc. The vehicle information of the user includes information related to the license plate number of the vehicle. The vehicle information may include information related to the vehicle model, year, body shape (sedan, wagon, minivan), body color, etc.

[0042] 5, the communication unit 30 performs bidirectional communication with the communication unit 13 of the photography system 2 via a network. The communication unit 30 transmits the license plate number of the target vehicle to the photography system 2. The communication unit 30 also receives from the photography system 2 an excerpted video CV0 including the target vehicle and feature amounts (driving state and appearance) of the target vehicle.

[0043] The object detection unit 34 performs object detection processing on each of the frames FRS0 to FRE0 of the cut-out video CV0. Fig. 7 shows an example of a frame on which the object detection processing has been performed.

[0044] In this frame, a road 61, a mountain 62, a tree 63, a pedestrian 64, and a plurality of vehicles 9A and 9B traveling on the road 61 are captured.

[0045] Region R62 indicates the region occupied by mountain 62. Region R63 indicates the region occupied by tree 63. Region R64 indicates the region occupied by pedestrian 64. Region RA indicates the region occupied by vehicle 9A, and region RB indicates the region occupied by vehicle 9B. Note that an object detection model such as YOLO (You Look Only Once) can be used for the "object detection process."

[0046] 5, the object detection unit 34 creates a detection video CV1 by performing object detection processing on all frames of the cut-out video CV0. For this reason, areas RA, RB, etc. are set in each frame FRS1 to FRE1 of the detection video CV1. Then, the object detection unit 34 transmits the detection video CV1 to the target vehicle identification unit 35.

[0047] The target vehicle identification unit 35 identifies a target vehicle in each of the frames FRS1 to FRE1 received from the object detection unit 34. The target vehicle identification unit 35 identifies the target vehicle from among the objects extracted by the object detection unit 34 based on the feature amount of the target vehicle (i.e., the running state such as running speed and acceleration, and the appearance such as the body shape and body color). This process is also referred to as a "target vehicle identification process." A trained model generated by a machine learning technique such as deep learning can also be used for the target vehicle identification process. In this example, the target vehicle identification unit 35 is realized by a "target vehicle identification model." In this embodiment, the target vehicle is vehicle 9A.

[0048] Then, a specific moving image CV2 in which the target vehicle is identified by the target vehicle identification unit 35 is generated. Note that the specific moving image CV2 includes a plurality of frames FRS2 to FRE2.

[0049] The frame extraction unit sequentially selects frames from the specific video CV2 and transmits them to the extraction processing unit 49. The extraction processing unit 49 includes an image processing unit 37 and an evaluation unit .

[0050] The extraction processing unit 49 executes an extraction process for extracting an extraction image from a selected frame selected by the frame extraction unit 36 ​​based on an evaluation value, which will be described later.

[0051] The extraction processing unit 49 generates an extracted image for each of all frames FRS2 to FRE2 and outputs it to the album creation unit 39. Then, the album creation unit 39 selects an extracted image having a high evaluation value from among the extracted images extracted from each selected frame as a highly evaluated image. The album creation unit 39 transmits the selected highly evaluated image to the output terminal 4 through the communication unit 30.

[0052] FIG. 8 is a flow diagram showing a flow from when the specific video CV2 is generated to when a highly rated image is selected.

[0053] The frame extraction unit 36 ​​transmits one untransmitted frame of the specific video CV2 as a selected frame SF to the extraction processing unit 49 (S100). The frame extraction unit 36 ​​first transmits frame FRS2. When the extraction processing unit 49 receives the selected frame SF, it performs extraction processing (S200).

[0054] When the extraction processing unit 49 ends the extraction processing, it transmits the extracted image to the album creation unit 39 (S300). The extraction processing unit 49 transmits a request signal to the frame extraction unit 36 ​​(S400). When the frame extraction unit 36 ​​receives the request signal, it determines whether all frames FRS2 to FRE2 have been transmitted (S410). When the frame extraction unit 36 ​​determines that all frames FRS2 to FRE2 have been transmitted to the extraction processing unit 49 (Yes in S410), it transmits an end signal to the extraction processing unit 49 (S420).

[0055] On the other hand, if the frame extraction unit 36 ​​determines that there is an untransmitted frame (No in S410), the process returns to S100 and transmits the untransmitted frame to the extraction processing unit 49.

[0056] Then, when extraction processing section 49 receives the end signal, extraction processing section 49 transmits a completion signal to frame extraction section 36 (S500).

[0057] The album creation section 39 selects a predetermined number of extracted images as highly rated images from the plurality of extracted images received (S600). The album creation section 39 selects a predetermined number of extracted images as highly rated images, for example, in descending order of evaluation value, which will be described later. Then, the album creation section 39 transmits the highly rated images to the output terminal 4 (S700).

[0058] FIG. 9 is a flow diagram showing the extraction process in S200. When performing the extraction process, first, the image processing unit 37 extracts the position of the vehicle 9A (S205). For example, the image processing unit 37 extracts the center position of the area RA from the selection frame SF. The image processing unit 37 sets an initial trimming range TR0 in the selection frame SF based on the center position of the area RA (S210). Note that the initial trimming range TR0 differs depending on the center position of the area RA.

[0059] For example, when the vehicle 9A is located at a position as shown in Fig. 10, the initial trimming range TR0 is set to an initial trimming range TR0(1). When the vehicle 9A is located at a position as shown in Fig. 11, the initial trimming range TR0 is set to an initial trimming range TR0(2).

[0060] In the example shown in Fig. 10, by setting the initial trimming range TR0(1), the positional relationship between the vehicle 9A, the mountain 62, and the tree 63 becomes a positional relationship that conforms to the rule of thirds. Also, in the example shown in Fig. 11, by setting the initial trimming range TR0(2), the positional relationship between the tree 63 and the vehicle 9A becomes a positional relationship that conforms to the rule of Hinomaru.

[0061] The division of thirds composition line 50 in FIG. 10 is a schematic representation of the division of thirds composition. The division-of-thirds composition line 50 divides the initial trimming range TR0 into nine parts. The division-of-thirds composition line 50 includes equally spaced vertical lines 51 and 52, equally spaced horizontal lines 53 and 54, an outer frame 55, and intersections P1, P2, P3, and P4. The outer frame 55 coincides with the initial trimming range TR0.

[0062] The initial trimming range TR0 is positioned so that the center position OA of the region RA coincides with position P4. The initial trimming range TR0 is set so that the apex O62 of the mountain 62 coincides with or is close to position P1. Furthermore, if the region between the vertical line 52 and the outer frame 55 is region R2, the initial trimming range TR0 is set so that region R63 is located within region R2, or so that most of region R63 is located within region R2.

[0063] Since the photographing area R1 is fixed, the positions of the vertex O62 and the tree 63 are fixed. Therefore, by extracting the center position OA, the image processing unit 37 can set the initial trimming range TR0 within the selected frame SF.

[0064] Returning to FIG. 9, the image processing unit 37 transmits the selected frame SF in which the initial trimming range TR0 has been set to the evaluation unit 38 (S215).

[0065] The evaluation unit 38 calculates an evaluation value EV of the initial trimming image TI0 of the initial trimming range TR0 (S220).

[0066] The evaluation unit 38 calculates an evaluation value EV0 of the initial trimmed image TI0 based on the evaluation items EI1 to EI5 and the deduction item DI1.

[0067] The evaluation unit 38 calculates the evaluation value EV based on the total value TP of the item scores IP1 to IP5 of the evaluation items EI1 to EI5 and the item coefficient DP of the deduction item DI1. Specifically, the evaluation value EV is calculated based on the following formula (1). Note that the item scores IP1 to IP5 are positive numbers.

[0068] Evaluation value EV = Total value TP × Item coefficient DP (1) Evaluation item EI1 is the positional relationship between the vehicle 9A, which is the subject to be imaged, and the mountain 62 and the tree 63, which are background subjects.

[0069] 10, the evaluation unit 38 calculates the item point IP1 of the evaluation item EI1 based on the amount of positional deviation between the intersection P1 and the vertex O62 and the amount of protrusion of the region R63 (region R63) from the region R2. The evaluation unit 38 lowers the item point IP1 as the amount of positional deviation between the intersection P1 and the vertex O62 increases, and lowers the item point IP1 as the amount of protrusion of the region R63 increases.

[0070] The evaluation item EI2 is the size of the vehicle 9A within the trimming range. The evaluation unit 38 calculates an item score IP2 of the evaluation item EI2 based on the area ratio AR0 of the region RA to the area of ​​the initial trimming range TR0. If the area ratio AR0 is not within a predetermined range, the evaluation unit 38 sets the item score IP2 low.

[0071] The evaluation item EI3 is the sharpness of the target vehicle and background objects. In the example shown in Fig. 10, the target vehicle is a vehicle 9A, and the background objects are a mountain 62 and a tree 63. The evaluation unit 38 calculates the sharpness of the images of the vehicle 9A, the mountain 62, the tree 63, etc. based on each pixel data of the initial trimmed image TI0, and calculates an item score IP3 of the evaluation item EI3.

[0072] The evaluation item EI4 is the brightness of the trimmed image. The evaluation unit 38 calculates the brightness of the initial trimmed image TI0 based on each pixel data of the initial trimmed image TI0, and calculates an item score IP4 of the evaluation item EI4. If the brightness of the initial trimmed image TI0 is not within a predetermined range, the evaluation unit 38 sets the item score IP4 low.

[0073] The evaluation item EI5 is the degree of cut-off of the target vehicle. The evaluation unit 38 calculates the degree of cut-off of the vehicle 9A based on the amount of the area RA protruding from the photographing area R1. The evaluation unit 38 lowers the item score IP5 of the evaluation item EI5 as the amount of the area RA protruding from the photographing area R1 increases.

[0074] The deduction item DI1 is whether or not an excluded object is present within the trimming range. If an excluded object is present within the trimming image, the evaluation unit 38 sets the item coefficient DP of the deduction item DI1 to "0", and if an excluded object is not present within the trimming image, the item coefficient DP is set to "1". The excluded object includes information that can identify an individual and vehicles other than the target vehicle. The information that can identify an individual includes, for example, a portrait whose face can be identified and a nameplate of a house. The information on the excluded object is stored in, for example, the storage unit 31. The evaluation unit 38 may calculate an evaluation value based on the above evaluation items EI1 to EI5.

[0075] In the example shown in FIG. 10, since the region RB falls within the initial trimming range TR0, the evaluation unit 38 sets the item coefficient DP of the deduction item DI1 related to the initial trimming image TI0 to "0."

[0076] In FIG. 9, when the evaluation unit 38 finishes the evaluation of the evaluation items EI1 to EI5 and the deduction item DI1, it calculates the evaluation value EV0 of the initial trimmed image TI0 (S220).

[0077] When the evaluation unit 38 finishes calculating the evaluation value, it transmits the frame FR in which the initial trimming range TR0 is set and the evaluation information EI to the storage unit 31 (S225).

[0078] After calculating the evaluation value EV0, the evaluation unit 38 determines whether it is greater than the threshold value Th (S230). The initial value of the threshold value Th is a negative value. Since the evaluation value EV0 is equal to or greater than 0, the evaluation value EV0 is greater than the threshold value Th.

[0079] When the evaluation unit 38 determines that the evaluation value EV0 is greater than the threshold value Th (Yes in S230), it sets reference evaluation information BEI (S235).

[0080] The reference evaluation information BEI is information on the trimmed image HTI with the highest evaluation value among the trimmed images already evaluated. The reference evaluation information BEI includes the trimmed image HTI, evaluation information EI of the trimmed image HTI, and information indicating the trimmed area HTR of the trimmed image HTI. The evaluation information EI may include information indicating each item point IP1 to IP5 of the trimmed image HTI, an item coefficient DP of the deduction item DI1, and an evaluation value HEV of the trimmed image HTI.

[0081] Furthermore, the reference evaluation information BEI may include the amount of positional deviation between the intersection P1 and the vertex O62 in the cropped image HTI, the amount of protrusion of the region R63 from the region R2, the area ratio AR, the sharpness of the target vehicle and background objects, the brightness of the cropped image, and the amount of protrusion of the region RA from the shooting area R1.

[0082] When the evaluation unit 38 evaluates only the initial trimmed image TI0, the reference evaluation information BEI includes the evaluation information EI of the initial trimmed image TI0 and information indicating the initial trimming range TR0. The trimmed image HTI is the initial trimming range TR0, and the trimming region HTR is the initial trimming range TR0.

[0083] When the evaluation unit 38 sets the reference evaluation information BEI, the evaluation unit 38 sets the threshold value Th as the evaluation value HEV (S240).

[0084] The evaluation unit 38 judges whether the update frequency of the reference evaluation information BEI is greater than a predetermined frequency (S245). The update frequency of the reference evaluation information BEI indicates how often the reference evaluation information BEI is updated within a predetermined time. Note that, in a state in which the evaluation unit 38 has evaluated only the initial trimmed image TI0, the update frequency of the reference evaluation information BEI is greater than the predetermined frequency.

[0085] If the evaluation unit 38 determines that the update frequency is greater than the predetermined frequency (Yes in S245), the evaluation unit 38 transmits the reference evaluation information BEI to the image processing unit 37 (S246). Upon receiving the reference evaluation information BEI, the image processing unit 37 updates the trimming range so that the evaluation value is higher than the evaluation value included in the reference evaluation information BEI (S250).

[0086] For example, when the image processing unit 37 determines that the item coefficient DP is "0", it deforms the initial cropping range TR0 so that the object to be excluded does not enter the cropping range, and sets the cropping range TR1.

[0087] Fig. 12 is a diagram showing the selected frame SF in a state where the trimming range TR1 has been set. In the state shown in Fig. 10, if the trimming range is shifted to the right so that the region RB does not fall within the initial trimming range TR0, a new region R64 will be included in the trimming range. Therefore, the evaluation unit 38 sets the trimming range TR1 so that the region R64 and the region RB do not fall within the trimming range.

[0088] Furthermore, the image processing unit 37 may set the trimming range TR1 so as not to include the regions RB and R64, and so as to improve at least one of the item scores IP1 to IP5 of the evaluation items EI1 to EI5.

[0089] For example, when the image processing unit 37 determines that the area ratio AR in the trimmed image HTI is larger than a predetermined range, it sets the trimming range TR1 longer in the vertical direction than the trimming area HTR. For example, when the image processing unit 37 determines that the area ratio AR is smaller than a predetermined range, it sets the trimming range TR1 shorter in the vertical direction than the trimming area HTR.

[0090] For example, when the image processing unit 37 determines that the amount of protrusion from the region R2 to the region R63 in the trimmed image HTI is greater than a predetermined value, it sets the trimming range TR1 so that the amount of protrusion is reduced.

[0091] For example, when the image processing unit 37 determines that the amount of positional deviation between the intersection point P1 and the vertex O62 in the trimmed image HTI is greater than a predetermined threshold, it sets the trimming range TR1 so as to reduce the amount of positional deviation.

[0092] Specifically, the image processor 37 sets the third division composition line 50A within the trimming range TR1. Then, the trimming range TR1 is set so that the distance between the intersection point P1A and the vertex O62 is smaller than the distance between the intersection point P1 and the vertex O62 in the trimming region HTR.

[0093] For example, when the image processing unit 37 determines that the sharpness of the vehicle 9A or the like in the trimmed image HTI is smaller than a predetermined threshold value, it performs edge enhancement processing or the like within the trimming range TR1.

[0094] For example, when the image processing unit 37 determines that the brightness of the trimmed image HTI is smaller than a predetermined threshold value, it increases the brightness of each pixel within the trimming range TR1.

[0095] Returning to FIG. 9, when the image processing unit 37 updates the trimming range to the trimming range TR1 (S250), the image processing unit 37 transmits the frame FR in which the trimming range TR1 has been set to the evaluation unit 38 (S215).

[0096] The evaluation unit 38 evaluates the trimmed image TI1 within the trimming range TR1 (S220). Specifically, the evaluation items EI1 to EI5 and the deduction item DI1 are evaluated to calculate an evaluation value EV1.

[0097] When the evaluation unit 38 completes the evaluation of the trimmed image TI1, it transmits the frame FR in which the trimming range TR1 is set and the evaluation information EI to the storage unit 31 (S225).

[0098] Then, in S230, the evaluation unit 38 determines whether the evaluation value EV1 is greater than the threshold value Th. If the evaluation value EV1 is greater than the threshold value Th (Yes in S230), the evaluation unit 38 resets the reference evaluation information BEI (S235).

[0099] On the other hand, if the evaluation unit 38 determines that the evaluation value EV1 is equal to or less than the threshold value Th (No in S230), it does not reset the reference evaluation information BEI, but determines whether the update frequency of the reference evaluation information BEI is greater than a predetermined frequency (S245).

[0100] Then, when the evaluation unit 38 determines that the update frequency of the reference evaluation information BEI is equal to or less than a predetermined frequency (No in S245), the evaluation unit 38 sets the trimmed image HTI set as the reference evaluation information BEI as the extracted image (S255).

[0101] The extraction process ends. Then, in Fig. 8, the evaluation section 38 of the extraction processing section 49 transmits the extracted image to the album creating section 39 (S300).

[0102] At this time, the extraction processing section 49 transmits to the album creating section 39 the trimmed image HTI as the extracted image, the evaluation value HEV, and central position information of the area RA within the trimmed image HTI.

[0103] Then, the frame extraction unit 36 ​​selects a predetermined number of extracted images from the received extracted images as highly evaluated images (S600). At this time, the highly evaluated images may be selected such that the centers of the areas RA of the extracted images are spaced apart from each other.

[0104] In addition, the extraction process in S200 may be performed using a trained model. <Pre-trained model> 13 is a diagram for explaining an example of a trained model (extraction processing model) used in the extraction processing. The estimation model 510, which is a pre-learning model, includes, for example, a neural network 511 and parameters 512. The neural network 511 is a known neural network used in image recognition processing by deep learning. Examples of such neural networks include a convolution neural network (CNN) and a recurrent neural network (RNN). The parameters 512 include weighting coefficients used in the calculation by the neural network 511.

[0105] A large amount of training data is prepared in advance by the developer. The training data includes example data and correct answer data. The example data is image data including a vehicle to be imaged. The correct answer data includes an extraction result corresponding to the example data. Specifically, the correct answer data is a trimmed image including the vehicle included in the example data and the background. Note that this trimmed image is a trimmed image having the above-mentioned evaluation value EV equal to or greater than a predetermined value.

[0106] The learning system 610 uses example data and correct answer data to train the estimation model 510. The learning system 610 includes an input unit 611, an extraction unit 612, and a learning unit 613.

[0107] The input unit 611 receives a large number of example data (image data) prepared by a developer and outputs the data to the extraction unit 612 .

[0108] The extraction unit 612 extracts vehicles included in the example data for each example data by inputting the example data from the input unit 611 to the estimation model 510. The extraction unit 612 outputs the extraction result (output from the estimation model 510) to the learning unit 613.

[0109] The learning unit 613 trains the estimation model 510 based on the vehicle extraction result from the example data received from the extraction unit 612 and the correct answer data corresponding to the example data. Specifically, the learning unit 613 adjusts the parameters 512 (e.g., weighting coefficients) so that the extraction result of the trimmed image including the vehicle and background obtained by the extraction unit 612 approaches the correct answer data.

[0110] As described above, the estimation model 510 is trained, and the trained estimation model 510 is stored in the extraction processing unit 49 as the extraction processing model 700. The extraction processing model 700 receives the specific video CV2 as an input, and outputs an extracted image including a vehicle and a background.

[0111] In this embodiment, at least one of the memory unit 19 and the memory unit 31 corresponds to the "memory unit" recited in the claims, and the calculation processing unit 32 corresponds to the "processing unit" recited in the claims.

[0112] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is defined by the claims rather than the description of the embodiments described above, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0113] 1 image processing system, 2 photography system, 3 server, 4 output terminal, 5 network, 9A, 9B vehicle, 10 identification camera, 11 viewing camera, 12 control device, 13 communication unit, 14 vehicle extraction unit, 15 number recognition unit, 16 matching processing unit, 17 target vehicle selection unit, 18 feature extraction unit, 19, 31 memory unit, 20 video extraction unit, 611 input unit, 32 calculation processing unit, 34 object detection unit, 35 target vehicle identification unit, 36 frame extraction unit, 37 image processing unit, 38 evaluation unit, 39 album creation unit, 40 web service management unit, 41 photography system management unit, 49 extraction processing unit, 50, 50A third division composition line, 51, 52 vertical line, 53, 54 horizontal line, 55 outer frame, 60, 61 road, 62 mountain, 63 tree, 64 Pedestrian, 81,82 license plate number, 301 image storage unit, 302 registration information storage unit, 510 estimation model, 511 neural network, 512 parameters, 610 learning system, 612 extraction unit, 613 learning unit, 700 extraction processing model, 2009 patent, AR,AR0 area ratio, BEI standard evaluation information, CV0 extracted video, CV1 detected video, CV2 specific video, DI1 deduction item, DP item coefficient, EI evaluation information, EI1,EI2,EI3,EI4,EI5 evaluation items, EV,EV0,EV1,HEV evaluation value.

Claims

1. A storage unit in which a plurality of frames captured by an imaging device for capturing moving images are stored; a processing unit that selects at least one frame from the plurality of frames stored in the storage unit, and outputs an extracted image extracted from the selected frame; Equipped with The processing unit includes: Set the initial trimming range to the frames you selected, if it is possible to set an updated cropping range that has a higher evaluation value than the image in the initial cropping range, an image cropped in the updated cropping range is output as the extracted image; if it is not possible to set a cropping range higher than the evaluation value of the image in the initial cropping range, the image in the initial cropping range is output as the extracted image; The evaluation value of the image in the cropping range is A positional relationship of an object within the image of the trimming range; The size of the object in the image of the trimming range; and The sharpness of the object in the image of the trimming range; The brightness of the image in the trimming range; A degree of cut-off of an object in the image of the trimming range; is set based on at least one evaluation item of the processing unit selects a first selection frame and a second selection frame from the plurality of frames; When an evaluation value of a first extracted image extracted from the first selected frame is higher than an evaluation value of a second extracted image extracted from the second selected frame, the image processing device outputs the first extracted image.

2. The image processing device according to claim 1 , wherein the processing unit lowers the evaluation value when a previously registered object to be excluded is captured within the image of the trimming range.

3. The image processing device according to claim 1 , wherein the object is a vehicle.

4. The image processing device according to claim 1 , wherein the processing unit sets an initial trimming range having a different composition based on a position of a vehicle as the object.

5. The processing unit acquires in advance at least one of license plate information of a target vehicle as the target object and appearance information that specifies an appearance of the target vehicle, 4. The image processing device according to claim 1, wherein the processing unit selects a frame in which the target vehicle is imaged from the plurality of frames, and creates the extracted image by trimming the selected frame so as to include the target vehicle.

6. Further comprising a memory in which an extraction processing model is stored; The image processing device according to claim 1 , wherein the extraction processing model is a trained model that receives an image including an object to be imaged as an input and outputs the extracted image having an evaluation value equal to or greater than a predetermined value.

Images