Image Processing System

Abstract

To provide an image processing system capable of acquiring an image of a vehicle being traveling, as an attractive one.SOLUTION: The image processing system comprises: a memory for storing moving image data obtained by photographing by a camera 32; and a processor which performs image processing on the moving image data stored in the memory. The processor extracts a frame acquired by photographing an object vehicle registered in advance out of the moving image photographed by the camera 32, identifies an object range occupied by the object vehicle and a specific range outside the object range, in the selected frame, trims the frame selected so as to include the object range on the basis of the object range and the specific range, and outputs an image.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to an image processing system. [Background technology]

[0002] A user who likes driving may have a desire (need) to take pictures of the exterior of his / her vehicle while driving. The user can upload the captured image to, for example, a social networking service (hereinafter, referred to as "SNS") so that many people can see it. However, it is difficult for a user to take a picture of the exterior of a vehicle while driving. Therefore, a service for taking pictures of the exterior of a vehicle while driving has been proposed. For example, JP 2019-121319 A discloses a vehicle photography support device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-121319 A Summary of the Invention [Problem to be solved by the invention]

[0004] The vehicle photography support device described in JP 2019-121319 A includes an identification unit 41 that identifies an external camera 20 capable of photographing the vehicle from the outside while it is moving, an instruction unit 43 that instructs the external camera 20 identified by the identification unit 41 to photograph the vehicle while it is moving, and an acquisition unit 44 that acquires the captured image obtained by the external camera 20 in accordance with the instruction from the instruction unit 43.

[0005] Furthermore, the vehicle imaging support device further includes a reception unit that receives imaging request information input by a user, and the imaging request information includes, for example, information regarding an editing pattern designated by the user.

[0006] However, the user cannot know in advance what kind of image will be captured by the external camera, and there is a possibility that the editing pattern specified by the user will not result in an attractive image.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and has an object to provide an image processing system that can acquire attractive images of a moving vehicle. [Means for solving the problem]

[0008] The image processing system according to the present disclosure includes a memory for storing video data captured by a camera, and a processor for performing image processing on the video data stored in the memory. The processor extracts frames in which a pre-registered target vehicle is captured from the video captured by the camera, identifies, in the selected frames, a target range occupied by the target vehicle and a specific range outside the target range, and based on the target range and the specific range, trims the selected frames to include the target range and outputs an image.

[0009] In the above image processing system, the specific range includes an exclusion range that is excluded from the range to be trimmed, and an inclusion range that is included in the range to be trimmed.

[0010] In the image processing system, the processor acquires in advance information that specifies the incorporation range.

[0011] In the image processing system, the processor performs trimming so that the target range is in a predetermined composition position.

[0012] The image processing system further includes a second model storage memory in which a trimming model is stored, and the trimming model is a learned module that receives as input a frame in which a target vehicle is imaged and outputs a trimmed image obtained by trimming an image from the input frame that includes the target vehicle and is composed in such a way that the image is not only composed as a whole, but also includes the target vehicle. Effect of the Invention

[0013] According to the image processing system of the present disclosure, it is possible to acquire attractive images of a vehicle in motion. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an overall configuration of an image processing system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a typical hardware configuration of an imaging system 1. [Diagram 3] FIG. 1 is a first diagram (perspective view) showing a state in which a vehicle is photographed by the photographing system 1. [Figure 4] 2 is a second diagram (top view) showing the state of vehicle photography by the photography system 1. FIG. [Diagram 5] FIG. 13 is a diagram showing an example of one frame of an identification video. [Figure 6] FIG. 1 is a diagram showing an example of one frame of a viewing video. [Figure 7] FIG. 2 is a block diagram showing a typical hardware configuration of the server 2. [Figure 8] 2 is a functional block diagram showing the functional configuration of the photography system 1 and the server 2. FIG. [Figure 9] 1 is a diagram showing specific positions P1, P2, and P3 in one frame of a moving image captured by appreciation moving image capture section 32. FIG. [Figure 10] In the extracted image IM1, the target vehicle is located at a specific position P1. [Figure 11] In the extracted image IM2, the target vehicle is located at a specific position P2. [Figure 12] In the extracted image IM3, the target vehicle is located at a specific position P3. [Figure 13] 1 is an extracted image IM1 that illustrates the results of analysis using an object detection model. [Figure 14] 13 is a diagram showing a schematic diagram of an extracted image IM1 in which a target range R1 and the like are shown and a three-division composition map CM1 is superimposed on the extracted image IM1. FIG. [Figure 15] FIG. 2 is a diagram showing a third division composition map CM1 and the like. [Figure 16] FIG. 13 shows a final image FIM1. [Figure 17] 1 is an extracted image IM2 that illustrates the results of analysis using the object detection model. [Figure 18] FIG. 13 is a diagram showing a state in which a Hinomaru composition map CM2 is superimposed on an extracted image IM2 in which a target range R1 and the like are shown. [Figure 19] FIG. 13 is a diagram showing a final image FIM2. [Figure 20] FIG. 13 is a diagram showing an extracted image IM3 analyzed using an object detection model and overlaid with a division-of-thirds composition map CM3. [Figure 21] FIG. 13 is a diagram showing a third division map CM3 and the like. [Figure 22] FIG. 13 shows a final image FIM3. [Figure 23] FIG. 1 is a diagram for explaining an example of a trained model (vehicle extraction model) used in the vehicle extraction process. [Figure 24] FIG. 1 is a diagram for explaining an example of a trained model (number recognition model) used in number recognition processing. [Diagram 25] FIG. 13 is a diagram for explaining an example of a trained model (target vehicle identification model) used in the target vehicle identification process. [Figure 26] FIG. 1 is a diagram for explaining an example of a trained model (frame extraction model) that extracts frames. [Figure 27] FIG. 13 is a diagram for explaining an example of a trained model (vehicle trimming model) for extracting a frame. [Figure 28] 5 is a flowchart showing a processing procedure of a vehicle photographing process in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference characters, and the description thereof will not be repeated.

[0016] [Embodiment Mode] <System configuration> Fig. 1 is a diagram showing an outline of the overall configuration of an image processing system according to the present embodiment. The image processing system 100 includes a plurality of photographing systems 1 and a server 2. Each of the plurality of photographing systems 1 and the server 2 are connected to each other via a network NW so as to be able to communicate with each other. Although three photographing systems 1 are shown in Fig. 1, the number of photographing systems 1 is not particularly limited. There may be only one photographing system 1.

[0017] The imaging system 1 is installed, for example, near a road and captures images of a vehicle 9 (see FIG. 3) traveling on the road. In this embodiment, the imaging system 1 performs a predetermined arithmetic processing (described later) on the captured video, and transmits the arithmetic processing result to the server 2 together with the video.

[0018] The server 2 is, for example, a server owned by a company that provides vehicle photography services. The server 2 may also be a cloud server provided by a cloud server management company. The server 2 generates images for the user to view from the video received from the photography system 1 (hereinafter, also referred to as "viewing images"), and provides the generated viewing images to the user. The viewing images are generally still images, but may be short videos. The user is often the driver of the vehicle 9, but is not particularly limited to this.

[0019] 2 is a block diagram showing a typical hardware configuration of the photography system 1. The photography system 1 includes a processor 11, a memory 12, a recognition camera 13, a viewing camera 14, and a communication interface (IF) 15. The memory 12 includes a ROM (Read Only Memory) 121, a RAM (Random Access Memory) 122, and a flash memory 123. The components of the photography system 1 are connected to each other via a bus or the like.

[0020] The processor 11 controls the overall operation of the imaging system 1. The memory 12 stores programs (operating system and application programs) executed by the processor 11 and data (maps, tables, formulas, parameters, etc.) used in the programs. The memory 12 also temporarily stores moving images captured by the imaging system 1.

[0021] Recognition camera 13 captures a video (hereinafter also referred to as an "identification video") for processor 11 to recognize the number on the license plate attached to vehicle 9. Viewing camera 14 captures a video (hereinafter also referred to as an "viewing video") used to generate a viewing image. Each of recognition camera 13 and viewing camera 14 is preferably a high-sensitivity camera with a polarized lens.

[0022] The communication IF 15 is an interface for communicating with the server 2. The communication IF 15 is, for example, a communication module that complies with 4G (Generation) or 5G.

[0023] FIG. 3 is a first diagram (perspective view) showing how a vehicle is photographed by the photographing system 1. FIG. 4 is a second diagram (top view) showing how a vehicle is photographed by the photographing system 1. With reference to FIGS. 3 and 4, the recognition camera 13 photographs the identification video from an angle (first angle) at which the license plate can be photographed. In this example, the identification video is photographed from approximately the front of the vehicle 9. Meanwhile, the viewing camera 14 photographs the viewing video from an angle (second angle) that looks good in photos (so-called SNS-friendly).

[0024] Fig. 5 is a diagram showing an example of one frame of an identification video. As shown in Fig. 5, multiple vehicles 9 (91, 92) may appear in the identification video. Hereinafter, the vehicle that is the subject of the shooting (the vehicle about to shoot the viewing image) among the multiple vehicles will be referred to as the "target vehicle" and will be distinguished from other vehicles.

[0025] 6 is a diagram showing an example of one frame of a viewing video. The viewing video is not required to show the license plate of the target vehicle. However, the license plate of the target vehicle may be shown in the viewing video.

[0026] In the example shown in FIG. 6, a human-shaped monument MO, a U-shaped road, a target vehicle TV and another vehicle OV traveling on the road are photographed.

[0027] The target vehicle TV and the other vehicles OV are not limited to four-wheeled vehicles as shown in Figures 3 to 6, but may be, for example, two-wheeled vehicles (motorcycles). Since the license plate of a motorcycle is only attached to the rear, it is easy for the license plate to not be photographed.

[0028] 7 is a block diagram showing a typical hardware configuration of the server 2. The server 2 includes a processor 21, a memory 22, an input device 23, a display 24, and a communication IF 25. The memory 22 includes a ROM 221, a RAM 222, and a HDD 223. The components of the server 2 are connected to each other via a bus or the like.

[0029] The processor 21 executes various types of arithmetic processing in the server 2. The memory 22 stores programs executed by the processor 21 and data used by the programs. The memory 22 also stores data used for image processing by the server 2, and stores data that has been image-processed by the server 2. The input device 23 accepts input from an administrator of the server 2. The input device 23 is typically a keyboard and a mouse. The display 24 displays various information. The communication IF 25 is an interface for communicating with the imaging system 1.

[0030] <Functional configuration of image processing system> 8 is a functional block diagram showing the functional configuration of the photography system 1 and the server 2. The photography system 1 includes an identification video photography unit 31, an appreciation video photography unit 32, a communication unit 33, and a calculation processing unit 34. The calculation processing unit 34 includes a vehicle extraction unit 341, a license plate recognition unit 342, a matching processing unit 343, a target vehicle selection unit 344, a feature extraction unit 345, a video buffer 346, and a video clipping unit 347.

[0031] The identification video shooting unit 31 shoots an identification video for the number recognition unit 342 to recognize the number on the license plate. The identification video shooting unit 31 outputs the identification video to the vehicle extraction unit 341. The identification video shooting unit 31 corresponds to the recognition camera 13 in FIG.

[0032] Appreciation video shooting unit 32 shoots an appreciation video for viewing by the user of vehicle 9. Appreciation video shooting unit 32 outputs the appreciation video to video buffer 346. Appreciation video shooting unit 32 corresponds to appreciation camera 14 in FIG.

[0033] The communication unit 33 performs two-way communication with a communication unit 42 (described later) of the server 2 via the network NW. The communication unit 33 receives the license plate number of the target vehicle from the server 2. The communication unit 33 also transmits the viewing video (more specifically, a video cut out from the viewing video so as to include the target vehicle) to the server 2. The communication unit 33 corresponds to the communication IF 15 in FIG. 2.

[0034] The vehicle extraction unit 341 extracts vehicles (not limited to the target vehicle, but vehicles in general) from the identification video. This process is also referred to as "vehicle extraction process." For the vehicle extraction process, for example, a trained model generated by machine learning technology such as deep learning can be used. In this example, the vehicle extraction unit 341 is realized by a "vehicle extraction model." The vehicle extraction model will be described in FIG. 23. The vehicle extraction unit 341 outputs videos from which vehicles have been extracted (frames including vehicles) from the identification video to the number recognition unit 342 and also to the matching processing unit 343.

[0035] The number recognition unit 342 recognizes the number of the license plate from the video in which the vehicle is extracted by the vehicle extraction unit 341. This process is also referred to as "number recognition process." A trained model generated by machine learning technology such as deep learning can also be used for the number recognition process. In this example, the number recognition unit 342 is realized by a "number recognition model." The number recognition model will be described in FIG. 24. The number recognition unit 342 outputs the recognized number to the matching processing unit 343. The number recognition unit 342 also outputs the recognized number to the communication unit 33. As a result, the number of each vehicle is transmitted to the server 2.

[0036] The matching processing unit 343 associates the vehicle extracted by the vehicle extraction unit 341 with the number recognized by the number recognition unit 342. This processing is also referred to as a "matching processing". Specifically, referring again to FIG. 5, a situation in which two vehicles 91, 92 are extracted and two numbers 81, 82 are recognized will be described as an example. The matching processing unit 343 calculates the distance between the number and the vehicle for each number (the distance between the coordinates of the number on the frame and the coordinates of the vehicle). Then, the matching processing unit 343 matches the number with a vehicle with a short distance between the number. In this example, the distance between the number 81 and the vehicle 91 is shorter than the distance between the number 81 and the vehicle 92, so the matching processing unit 343 associates the number 81 with the vehicle 91. Similarly, the matching processing unit 343 associates the number 82 with the vehicle 92. The matching processing unit 343 outputs the result of the matching processing (the vehicle with the associated number) to the target vehicle selection unit 344.

[0037] The target vehicle selection unit 344 selects, as a target vehicle, a vehicle whose number matches the number of the target vehicle (received from the server 2) from among the vehicles whose numbers have been associated by the matching process. The target vehicle selection unit 344 outputs the vehicle selected as the target vehicle to the feature extraction unit 345.

[0038] The feature extraction unit 345 extracts the feature of the target vehicle by analyzing a video including the target vehicle. More specifically, the feature extraction unit 345 calculates the running speed of the target vehicle based on the temporal change of the target vehicle in the frames including the target vehicle (for example, the movement amount of the target vehicle between frames, the change amount of the size of the target vehicle between frames). In addition to the running speed of the target vehicle, the feature extraction unit 345 may calculate, for example, the acceleration (deceleration) of the target vehicle. Furthermore, the feature extraction unit 345 extracts information on the appearance (body shape, body color, etc.) of the target vehicle using a known image defect recognition technology. The feature extraction unit 345 outputs the feature (running state and appearance) of the target vehicle to the video cutout unit. Furthermore, the feature extraction unit 345 outputs the feature of the target vehicle to the communication unit 33. As a result, the feature of the target vehicle is transmitted to the server 2.

[0039] Video buffer 346 temporarily stores viewing videos. Video buffer 346 is typically a ring buffer (circulatory buffer) and has a circular storage area in which the beginning and end of a one-dimensional array are logically linked. Newly captured viewing videos are stored in video buffer 346 for a predetermined time that can be stored in the storage area. Viewing videos (old videos) that exceed the predetermined time are automatically deleted from video buffer 346.

[0040] The video clipping unit 347 clips out a portion of the viewing video stored in the video buffer 346 that is likely to include the target vehicle, based on the feature values ​​(traveling speed, acceleration, body shape, body color, etc., of the target vehicle) extracted by the feature value extraction unit 345. To explain in more detail, the distance between the point photographed by the identification video shooting unit 31 (recognition camera 13) and the point photographed by the viewing video shooting unit 32 (viewing camera 14) is known. Therefore, if the traveling speed (and acceleration) of the target vehicle is known, the video clipping unit 347 can calculate the time difference between the timing at which the target vehicle is photographed by the identification video shooting unit 31 and the timing at which the target vehicle is photographed by the viewing video shooting unit 32. The video clipping unit 347 calculates the timing at which the target vehicle is photographed by the viewing video shooting unit 32, based on the timing at which the target vehicle is photographed by the identification video shooting unit 31 and the above-mentioned time difference. Then, the video clipping unit 347 clips out a video of a predetermined time width (for example, several seconds to several tens of seconds) including the timing when the target vehicle is photographed from the viewing video stored in the video buffer 346. The video clipping unit 347 outputs the clipped viewing video to the communication unit 33. As a result, the viewing video including the target vehicle is transmitted to the server 2.

[0041] The video clipping unit 347 may clip the viewing video at a predetermined timing, regardless of the feature extracted by the feature extraction unit 345. In other words, the video clipping unit 347 may clip the viewing video captured by the viewing video capturing unit 32 after a predetermined time difference from the timing when the target vehicle is captured by the identification video capturing unit 31.

[0042] The server 2 includes a storage unit 41, a communication unit 42, and a calculation processing unit 43. The storage unit 41 includes an image storage unit 411 and a registration information storage unit 412. The calculation processing unit 43 includes a vehicle extraction unit 431, a target vehicle identification unit 432, a frame extraction unit 433A, an image processing unit 433B, an album creation unit 434, a web service management unit 435, and an image capture system management unit 436.

[0043] The image storage unit 411 stores the final image obtained as a result of the calculation processing by the server 2. More specifically, the image storage unit 411 stores images before and after processing by the frame extraction unit 433A and the image processing unit 433B, and also stores an album created by the album creation unit 434.

[0044] The registration information storage unit 412 stores registration information related to the vehicle photography service. The registration information includes personal information of a user who has applied for the vehicle photography service and vehicle information of the user. The personal information of the user includes information related to the user's identification number (ID), name, date of birth, address, telephone number, email address, etc. The vehicle information of the user includes information related to the license plate number of the vehicle. The vehicle information may include information related to the vehicle model, year, body shape (sedan, wagon, minivan), body color, etc.

[0045] The communication unit 42 performs bidirectional communication with the communication unit 33 of the photography system 1 via the network NW. The communication unit 42 transmits the license plate number of the target vehicle to the photography system 1. The communication unit 42 also receives a viewing video including the target vehicle and feature amounts (driving state and appearance) of the target vehicle from the photography system 1. The communication unit 42 corresponds to the communication IF 25 in FIG. 7.

[0046] The vehicle extraction unit 431 extracts vehicles (not limited to the target vehicle, but vehicles in general) from the viewed video. For this process, a vehicle extraction model can be used, similar to the vehicle extraction process by the vehicle extraction unit 341 of the imaging system 1. The vehicle extraction unit 431 outputs a video from which a vehicle has been extracted (a frame including a vehicle) to the target vehicle identification unit 432.

[0047] The target vehicle identification unit 432 identifies a target vehicle from among the vehicles extracted by the vehicle extraction unit 431 based on the feature quantities of the target vehicle (i.e., driving conditions such as driving speed and acceleration, and external appearances such as body shape and body color). This process is also referred to as "target vehicle identification process." A trained model generated by machine learning techniques such as deep learning can also be used for the target vehicle identification process. In this example, the target vehicle identification unit 432 is realized by a "target vehicle identification model." Target vehicle identification will be described in FIG. 25. At least one viewing image is generated by the target vehicle identification unit 432 identifying the target vehicle. The viewing image usually includes multiple images (multiple frames that are consecutive in time).

[0048] The frame extraction unit 433A extracts an image (frame) in which the target vehicle is located at a predetermined specific position from the viewing image output from the target vehicle identification unit 432. This process is also referred to as a frame extraction process. Note that the specific position is not limited to one, and may be set at multiple locations.

[0049] 9 is a diagram showing specific positions P1, P2, and P3 in one frame of a moving image captured by the appreciation moving image capture section 32. In this frame, a plurality of specific positions P1, P2, and P3 are set.

[0050] The imaging range R0 of the appreciation moving image imaging section 32 is fixed, and the specific positions P1, P2, and P3 are positions that are determined in advance within the imaging range R0.

[0051] The viewing image output by vehicle identification unit 432 to frame extraction unit 433 includes image data for each frame, and information indicating the position information of the target vehicle in each frame and the area occupied by the target vehicle. Frame extraction unit 433A determines whether the target vehicle is located at any of specific positions P1, P2, P3 in each frame, based on the position information of the target vehicle included in each frame and the area occupied by the target vehicle.

[0052] The specific positions P1, P2, P3 can be determined, for example, by verifying in advance which position the target vehicle is positioned in to provide an aesthetically pleasing driving posture. In particular, when the imaging range R0 of the appreciation video imaging unit 32 is fixed, the driving posture of the target vehicle at the specific positions P1, P2, P3 can be grasped in advance, and therefore the specific positions P1, P2, P3 can be determined by verifying in advance. Note that there may be only one specific position.

[0053] A good driving posture includes a vehicle posture that gives a sense of dynamism when cornering and a vehicle posture that gives a sense of speed when driving on a straight line. A vehicle posture that gives a sense of dynamism when cornering includes the posture when entering a corner, the posture during cornering, and the posture when exiting a corner.

[0054] Fig. 10 is an extracted image IM1 in which a target vehicle is located at a specific position P1. Fig. 11 is an extracted image IM2 in which a target vehicle is located at a specific position P2. Fig. 12 is an extracted image IM3 in which a target vehicle is located at a specific position P3. In this embodiment, frame extraction unit 433A extracts extracted images IM1 to IM3.

[0055] In addition, in the frame extraction process, for example, a trained model (frame extraction model) generated by a machine learning technique such as deep learning can be used. The "frame extraction model" will be described later with reference to FIG. 26.

[0056] The extracted images IM1, IM2, and IM3 shown in FIGS. 10 to 12 include a target vehicle TV, another vehicle OV, and a monument MO.

[0057] Frame extraction section 433A outputs at least one extracted image (extracted frame) to image processing section 433B. In this embodiment, extraction images IM1, IM2, and IM3 are output to image processing section 433B.

[0058] The image processing unit 433B performs trimming on the extracted image so that the target vehicle TV is included in the extracted image. This processing is called vehicle trimming processing.

[0059] The vehicle trimming process determines the area occupied by the target vehicle TV in the extracted image and the area occupied by the surrounding area other than the target vehicle.

[0060] Then, for example, based on the division of thirds composition, the target vehicle TV and the surrounding image other than the target vehicle are trimmed from the extracted image. In this manner, the vehicle trimming process is performed, and the final image of the division of thirds composition can be obtained. Note that the composition method is not limited to the division of thirds composition method, and various composition methods such as the division of thirds composition method, the triangular composition method, and the Hinomaru composition method may be adopted.

[0061] When performing the vehicle trimming process, the background included in the surrounding image may be taken into consideration. For example, when the image processing unit 433B determines that the surrounding image includes an image of another vehicle OV, the image processing unit 433B performs the vehicle trimming process so that the other vehicle or other excluded object is not included.

[0062] In this case, the image processing unit 433B detects objects around the target vehicle TV using an object detection model such as YOLO (You Look Only Once). Then, the image processing unit 433B identifies excluded objects such as other vehicles OV, and performs vehicle trimming processing on the extracted image so that the excluded objects are not included.

[0063] When performing the vehicle trimming process, a specific background may be included in the surrounding image, for example, a specific background such as the sea or a mountain may be included in the background.

[0064] When the imaging range R0 of the appreciation moving image imaging section 32 is fixed, the position of the subject to be incorporated as the background (the incorporation subject) is fixed.

[0065] In this case, the image processing unit 433B acquires information indicating the position and range of the incorporation target in advance. Then, in the vehicle trimming process, the extracted image is trimmed so that the target vehicle TV and the incorporation target are included, the exclusion target is not included, and the target vehicle TV is positioned based on the rule of thirds or the like. Note that the object may be trimmed so that an object recognized as the incorporation target in the object detection model is included as the incorporation target.

[0066] FIG. 13 is an extracted image IM1 that shows a schematic representation of the results of an analysis using an object detection model. In FIG. 13, a target range R1 indicates the range occupied by a target vehicle TV. An excluded range R2 indicates the range occupied by other vehicles OV. An inclusion range R3 indicates the range occupied by a monument MO. Note that in FIG. 13, other vehicles OV are shown as an example of an excluded target, but the excluded target may include an image of an identifiable person, a nameplate on a house, and the like. Also, a monument MO is shown as an example of an included target, but it may also include, for example, a conspicuous building, a tree, and the like.

[0067] Fig. 14 is a diagram showing a state in which the three-division composition map CM1 is superimposed on the extraction image IM1 showing the target range R1 etc. Fig. 15 is a diagram showing the three-division composition map CM1 etc.

[0068] When the target vehicle TV is located at the specific position P1, the target vehicle TV faces diagonally forward, and therefore the image processing unit 433B employs the three-division composition map CM1.

[0069] The image processing unit 433B superimposes the three-division composition map CM1 on the extracted image IM1. The three-division composition map CM1 has a rectangular shape, and includes an outer line of the rectangular shape, vertical division lines L1, L2, and horizontal division lines L3, L4.

[0070] The image processing unit 433B adjusts the size of the third-division composition map CM1 so that the target range R1 and the inclusion range R3 are included in the third-division composition map CM1, but the exclusion range R2 is not included. Furthermore, the image processing unit 433B positions the third-division composition map CM1 so that the intersection P10 of the vertical division line L1 and the horizontal division line L3 is located within the target range R1. Then, the image processing unit 433B trims the extraction image IM1 along the outline of the third-division composition map CM1. In this way, the image processing unit 433B can create the extraction image IM1 by performing a vehicle trimming process on the extraction image IM1.

[0071] 16 is a diagram showing a final image FIM1. According to this final image FIM1, in a three-division composition, the target vehicle TV can be included in the image, and the monument MO can also be included in the image. In addition, in the final image FIM1, other vehicles OV can be excluded.

[0072] Although the above description is based on extracted image IM1, image processing unit 433B also applies vehicle trimming processing to extracted images IM2 and IM3.

[0073] 17 is an extracted image IM2 that shows a schematic diagram of the result of analysis using the object detection model. The image processing unit 433B also sets a target range R1, an excluded range R2, and an inclusion range R3 in the extracted image IM2.

[0074] 18 is a diagram showing a state in which a Hinomaru composition map CM2 is superimposed on an extracted image IM2 in which a target range R1 and the like are shown. The Hinomaru composition map CM2 includes a rectangular outline and a virtual circle CL. The virtual circle CL is located in the center of the Hinomaru composition map CM2.

[0075] The image processing unit 433B sets up the Hinomaru composition map CM2 so that the center of the target range R1 is located within the virtual circle CL.

[0076] In addition, when the target vehicle TV is located at the specific position P2, the target vehicle TV faces directly to the side. Therefore, the image processing unit 433B selects the Hinomaru composition map CM2 as the composition map. In this way, when the specific position is specified in advance, the composition map to be adopted is changed according to the specific position.

[0077] In Fig. 18, when the inclusion range R3 is to be included in the Hinomaru composition map CM2, the exclusion range R2 enters the Hinomaru composition map CM2, so the image processing unit 433B sets the Hinomaru composition map CM2 so that only the target range R1 is included in the Hinomaru composition map CM2. Then, the image processing unit 433B trims the extracted image IM2 along the outline of the Hinomaru composition map CM2. In this way, the image processing unit 433B performs vehicle trimming processing on the extracted image IM2, thereby creating the final image FIM2.

[0078] 19 is a diagram showing a schematic diagram of the final image FIM2. According to this final image FIM2, an image based on the Hinomaru composition can be obtained. Furthermore, an image that does not include other vehicles OV, which are to be excluded, can be obtained.

[0079] It should be noted that the image of the target vehicle TV may also be trimmed using the three-division composition map CM1 or the like in the extracted image IM2 of the vehicle located at the specific position P2.

[0080] FIG. 20 is a diagram showing an extracted image IM3 analyzed using an object detection model and overlaid with a three-division composition map CM3.

[0081] The image processing unit 433B selects the three-division composition map CM3 because the target vehicle TV located at the specific position P3 faces obliquely.

[0082] 21 is a diagram showing the third-division composition map CM3 etc. The third-division composition map CM3 is arranged so that the intersection point P11 where the vertical division line L1 and the horizontal division line L4 of the third-division composition map CM3 intersect overlaps with the target range R1.

[0083] Then, the image processing unit 433B adjusts the size of the third-division composition map CM1 so that the subject range R1 and the inclusion range R3 are included in the third-division composition map CM3, but the exclusion range R2 is not included. In the example shown in Fig. 21, if the third-division composition map CM3 is set so that the subject range R1 and the inclusion range R3 are included, the exclusion range R2 will be included. Therefore, the image processing unit 433B sets the third-division composition map CM3 so that only the subject range R1 is located within the third-division composition map CM3.

[0084] Then, the image processing unit 433B can create a final image FIM3 by trimming the extracted image IM3 along the outline of the triangular composition map CM3.

[0085] FIG. 22 is a diagram showing the final image FIM3. In this way, an image with a three-division composition can be obtained. The image processing unit 433B sets the trimming range so that the number of pixels in the trimming range is not smaller than a preset number of pixels. If the number of pixels becomes too small, the image of the target vehicle TV will become unclear in the final image. The trimming range is also set so that the proportion of the target range R1 in the trimming range is larger than a predetermined value. This is to prevent the target vehicle TV from becoming too small in the final image FIM3.

[0086] For the frame extraction process, a trained model (frame extraction model) generated by machine learning techniques such as deep learning can be used for the vehicle trimming process. The "vehicle trimming model" will be described later in FIG. 27.

[0087] Returning to FIG. 8, the image processing unit 433B outputs the final images FIM1 to FIM3 to the album creation unit 434. The album creation unit 434 creates an album using the final images. A known image analysis technique (for example, a technique for automatically creating a photo book, a slide show, or the like from images taken with a smartphone) can be used to create the album. The album creation unit 434 outputs the album to the web service management unit 435.

[0088] Web service management section 435 provides a web service (for example, an application program that can be linked to an SNS) using the album created by album creation section 434. Note that web service management section 435 may be implemented in a server different from server 2.

[0089] The photography system management unit 436 manages (monitors and diagnoses) the photography system 1. When any abnormality (camera failure, communication failure, etc.) occurs in the photography system 1 under its management, the photography system management unit 436 notifies the administrator of the server 2 of the occurrence of the abnormality. This enables the administrator to take measures such as inspecting and repairing the photography system 1. Like the web service management unit 435, the photography system management unit 436 can also be implemented as a separate server.

[0090] <Pre-trained model> 23 is a diagram for explaining an example of a trained model (vehicle extraction model) used in vehicle extraction processing. Estimation model 51, which is a pre-trained model, includes, for example, neural network 511 and parameters 512. Neural network 511 is a known neural network used in image recognition processing by deep learning. Examples of such neural networks include a convolution neural network (CNN) and a recurrent neural network (RNN). Parameters 512 include weighting coefficients used in calculations by neural network 511.

[0091] A large amount of training data is prepared in advance by the developer. The training data includes example data and correct answer data. The example data is image data including the vehicle to be extracted. The correct answer data includes an extraction result corresponding to the example data. Specifically, the correct answer data is image data from which the vehicle included in the example data has been extracted.

[0092] The learning system 61 uses example data and correct answer data to train the estimation model 51. The learning system 61 includes an input unit 611, an extraction unit 612, and a learning unit 613.

[0093] The input unit 611 receives a large number of example data (image data) prepared by a developer and outputs the data to the extraction unit 612 .

[0094] The extraction unit 612 extracts vehicles included in the example data for each example data by inputting the example data from the input unit 611 to the estimation model 51. The extraction unit 612 outputs the extraction result (output from the estimation model 51) to the learning unit 613.

[0095] The learning unit 613 trains the estimation model 51 based on the vehicle extraction result from the example data received from the extraction unit 612 and the correct answer data corresponding to the example data. Specifically, the learning unit 613 adjusts the parameters 512 (e.g., weighting coefficients) so that the vehicle extraction result obtained by the extraction unit 612 approaches the correct answer data.

[0096] As described above, the estimation model 51 is trained, and the trained estimation model 51 is stored in the vehicle extraction unit 341 (and the vehicle extraction unit 431) as the vehicle extraction model 71. The vehicle extraction model 71 receives an identification video as input, and outputs an identification video from which a vehicle is extracted. For each frame of the identification video, the vehicle extraction model 71 associates the extracted vehicle with the identifier of the frame and outputs the associated vehicle to the matching processing unit 343. The frame identifier is, for example, a timestamp (time information of the frame).

[0097] FIG. 24 is a diagram for explaining an example of a trained model (number recognition model) used in number recognition processing. The example data is image data including the number to be recognized. The correct answer data is data indicating the position and number of the license plate included in the example data. Although the example data and correct answer data are different, the learning method of the estimation model 52 by the learning system 62 is similar to the learning method by the learning system 61 (see FIG. 9), and therefore detailed description will not be repeated.

[0098] The estimation model 52 for which learning has been completed is stored in the license plate recognition unit 342 as the license plate recognition model 72. The license plate recognition model 72 receives as input the identification video in which a vehicle is extracted by the vehicle extraction unit 341, and outputs the coordinates and number of the license plate. For each frame of the identification video, the license plate recognition model 72 associates the recognized coordinates and number of the license plate with the identifier of the frame and outputs them to the matching processing unit 343.

[0099] FIG. 25 is a diagram for explaining an example of a trained model (target vehicle identification model) used in the target vehicle identification process. The example data is image data including a target vehicle that is an identified target. The example data further includes information regarding the features of the target vehicle (specifically, the driving state and appearance). The correct answer data is image data in which the target vehicle included in the example data is identified. The learning method of the estimation model 53 by the learning system 63 is similar to the learning method by the learning systems 61, 62 (see FIG. 23 and FIG. 24), so detailed description will not be repeated.

[0100] The estimation model 53 for which learning has been completed is stored in the target vehicle identification unit 432 as the target vehicle identification model 73. The target vehicle identification model 73 receives as input the viewed video in which a vehicle is extracted by the vehicle extraction unit 431 and the feature quantities (driving state and appearance) of the target vehicle, and outputs the viewed video in which the target vehicle is identified. For each frame of the viewed video, the target vehicle identification model 73 associates the identified viewed video with the identifier of the frame and outputs it to the frame extraction unit 433A.

[0101] The vehicle extraction process is not limited to the process using machine learning. Any known image recognition technology (image recognition model, algorithm) that does not use machine learning can be applied to the vehicle extraction process. The same applies to the license plate recognition process and the target vehicle identification process.

[0102] FIG. 26 is a diagram illustrating an example of a trained model (frame extraction model) that extracts a frame.

[0103] The example data is a plurality of image frames including a vehicle to be recognized. The correct answer data includes an extraction result corresponding to the example data. Specifically, the correct answer data is an image frame that shows a vehicle with a good driving posture from the plurality of image frames of the example data.

[0104] Although the example data and correct answer data are different, the learning method of estimation model 54 by learning system 64 is similar to the learning method by learning system 61 and the like, and therefore detailed description will not be repeated.

[0105] The estimation model 54 for which learning has been completed is stored in the frame extraction unit 433A as the frame extraction model 74. The frame extraction model 74 receives as input a viewing video in which a target vehicle is identified, and outputs a frame in which the target vehicle with a good driving posture is captured as an extracted image to the image processing unit 433B.

[0106] The frame extraction process is not limited to the process using machine learning. A known image recognition technique (image recognition model, algorithm) that does not use machine learning can be applied to the frame extraction process. FIG. 27 is a diagram for explaining an example of a trained model (vehicle trimming model) from which a frame is extracted.

[0107] The example data is a plurality of image frames including a vehicle to be recognized. The image frames of the example data preferably include at least one of an excluded object and an included object. The correct answer data is a cropped image cropped from the example data. Specifically, the correct answer data is a cropped image that includes the vehicle to be recognized and the included object, and is cropped so as to exclude the excluded object.

[0108] When the inclusion target and the exclusion target are included, the correct answer data includes an image that is cropped to include the vehicle to be recognized, but not to include the inclusion target and the exclusion target. Furthermore, the correct answer data includes an image in which a cropping range is set so that the number of pixels in the cropping range is a predetermined number or more. Note that the correct answer data includes an image in which the cropping range is set so that the range occupied by the vehicle to be recognized is a predetermined number or more. The cropped image of the correct answer data is a cropped image cropped using various compositions such as the division of thirds composition, the division of fourths composition, the triangular composition, and the Hinomaru composition.

[0109] Although the example data and correct answer data are different, the learning method of estimation model 54 by learning system 64 is similar to the learning method by learning system 61 and the like, and therefore detailed description will not be repeated.

[0110] The estimation model 52 for which the learning has been completed is stored as a vehicle trimming model 75 in the image processing unit 433B.

[0111] The vehicle trimming model 75 receives an extracted image (frame) of a target vehicle with a good driving posture as input, and outputs a well-composed trimmed image as a final image. Note that the vehicle trimming process is not limited to the process using machine learning. A known image recognition technology (image recognition model, algorithm) that does not use machine learning can be applied to the frame extraction process.

[0112] <Processing flow> Fig. 28 is a flowchart showing the procedure of the vehicle photography processing in this embodiment. This flowchart is executed, for example, when a predetermined condition is met or at a predetermined cycle. In the figure, the left side shows the processing by the photography system 1, and the right side shows the processing by the server 2. Each step is realized by software processing by the processor 11 or the processor 21, but may also be realized by hardware (electrical circuitry). Hereinafter, step is abbreviated as S.

[0113] In S11, the photography system 1 extracts a vehicle by executing a vehicle extraction process (see FIG. 9) on the identification video. Furthermore, the photography system 1 recognizes the license plate by executing a license plate recognition process (see FIG. 10) on the identification video from which the vehicle has been extracted (S12). The photography system 1 transmits the recognized license plate to the server 2.

[0114] When the server 2 receives the number from the photography system 1, it refers to the registration information to determine whether the received number is a registered number (i.e., whether the vehicle photographed by the photography system 1 is the vehicle (target vehicle) of the user who applied for the vehicle photography service). If the received number is a registered number (number of the corresponding vehicle), the server 2 transmits the number of the target vehicle and requests the photography system 1 to transmit a viewing video including the target vehicle (S21).

[0115] In S13, the photography system 1 executes a matching process between each vehicle in the recognition video and each license plate number. Then, the photography system 1 selects a vehicle having the same license plate number as the target vehicle as a corresponding vehicle from among the vehicles having license plate numbers (S14). Furthermore, the photography system 1 extracts the feature amount (driving state and appearance) of the target vehicle and transmits the extracted feature amount to the server 2.

[0116] In S16, the photography system 1 cuts out a portion including the target vehicle from the viewing video temporarily stored in the memory 22 (video buffer 346). In this cutting out, the driving state (driving speed, acceleration, etc.) and the appearance (body shape, body color, etc.) of the target vehicle can be used as described above. The photography system 1 transmits the cut-out viewing video to the server 2.

[0117] In S22, the server 2 executes a vehicle extraction process (see FIG. 9) on the viewing video received from the imaging system 1 to extract vehicles.

[0118] In S23, the server 2 identifies a target vehicle from among the vehicles extracted in S22 based on the feature amount (driving state and appearance) of the target vehicle (target vehicle identification process in FIG. 11). It is also possible to use only one of the driving state and appearance of the target vehicle as the feature amount of the target vehicle. However, there is a possibility that a plurality of vehicles having the same body shape and body color are included in the viewing video, or a plurality of vehicles having approximately the same driving speed and acceleration are included. In contrast, in this embodiment, even if a plurality of vehicles having the same body shape and body color are included in the viewing video, the target vehicle can be distinguished from other vehicles if the driving speed and / or acceleration are different between the vehicles. Alternatively, even if a plurality of vehicles having approximately the same driving speed and acceleration are included in the viewing video, the target vehicle can be distinguished from other vehicles if the body shape and / or body color are different between the vehicles. In this way, by using both the driving state and appearance of the target vehicle as the feature amount of the target vehicle, the identification accuracy of the target vehicle can be improved.

[0119] However, it is not essential to use both the driving condition and the appearance of the target vehicle, and only one of them may be used. Information on the driving condition and / or the appearance of the target vehicle corresponds to the "target vehicle information" according to the present disclosure. In addition, the information on the appearance of the target vehicle is not limited to vehicle information obtained by analysis by the imaging system 1 (feature amount extraction unit 345), and may be vehicle information previously stored in the registration information storage unit 412.

[0120] In S24, the server 2 extracts at least one extracted image (extracted frame) in which the target vehicle TV having a good driving posture is captured, from the viewing video (plurality of viewed images) including the target vehicle.

[0121] In S25, the server 2 trims the extracted image to include the target vehicle and to have a specified composition, and extracts a final image including the target vehicle. In S25, the trimming is performed so that the excluded target is not included. If the excluded target is not included, the server 2 trims to include the included target. If the included target would include the excluded target, the server 2 trims to exclude both the included target and the excluded target. The server 2 trims so that the number of pixels in the trimmed range is equal to or greater than a specified number. The server 2 trims so that the range occupied by the vehicle to be recognized within the trimmed range is equal to or greater than a specified number.

[0122] Then, the server 2 creates an album using the final images (S26). The user can view the created album and post desired images in the album to an SNS.

[0123] In the present embodiment, an example has been described in which the image processing is shared between the photographing system 1 and the server 2. Therefore, both the processor 11 of the photographing system 1 and the processor 21 of the server 2 correspond to the "processor" according to the present disclosure. However, the photographing system 1 may perform all image processing and transmit the image-processed data (viewed image) to the server 2. Therefore, the server 2 is not an essential component for the image processing according to the present disclosure. In this case, the processor 11 of the photographing system 1 corresponds to the "processor" according to the present disclosure. Alternatively, conversely, the photographing system 1 may transmit all captured videos to the server 2, and the server 2 may perform all image processing. In this case, the processor 21 of the server 2 corresponds to the "processor" according to the present disclosure.

[0124] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is defined by the claims rather than the description of the embodiments described above, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0125] 100 image processing system, 1,1A photography system, 11 processor, 12 memory, 121 ROM, 122 RAM, 123 flash memory, 13 camera, 14 communication IF, 15 communication IF, 151 long-distance wireless module, 152 short-distance wireless module, 2 server, 21 processor, 22 memory, 221 ROM, 222 RAM, 23 input device, 24 display, 25 communication IF, 31 photography unit, 32 communication unit, 33 calculation processing unit, 331 video buffer, 332 vehicle extraction unit, 333 number recognition unit, 334 matching processing unit, 335 target vehicle selection unit, 336 feature extraction unit, 337 video cutout unit, 41 memory unit, 411 image memory unit, 412 registration information memory unit, 42 communication unit, 43 calculation processing unit, 431 vehicle extraction unit, 432 Target vehicle identification unit, 433 Image processing unit, 434 Album creation unit, 435 Web service management unit, 436 Photography system management unit, 51, 52, 53 Estimation model, 511 Neural network, 512 Parameters, 61, 62, 63 Learning system, 611, 621, 631 Input unit, 612 Extraction unit, 622 Recognition unit, 632 Identification unit, 613, 623, 633 Learning unit, 71 Vehicle extraction model, 72 Number recognition model, 73 Target vehicle identification model, 81 Short-range communication unit, 82 Photography unit, 83 Long-range communication unit, 84 Calculation processing unit, 841 Radio ID acquisition unit, 842 Video buffer, 843 Vehicle extraction unit, 844 Matching processing unit, 845 Target vehicle selection unit, 846 Feature extraction unit, 847 Video cutout unit, 9 Vehicle, 93 Camera, 95 short-range wireless module, 96 user terminal, 900 image processing system, NW network.

Claims

[Claim 1] A memory for storing video data captured by a camera; a processor that performs image processing on the video data stored in the memory; Equipped with The processor, Extracting frames in which a pre-registered target vehicle is captured from the video captured by the camera; Identifying a target range in which the target vehicle occupies and a specific range other than the target range in the extracted frame; The specific range includes an exclusion range that is excluded from the range to be trimmed and an inclusion range that is included in the range to be trimmed, The processor acquires information for identifying the incorporation range in advance, The processor overlays a rule of thirds map onto the extracted frame; the third-division composition map includes a rectangular outer edge, first and second vertical division lines disposed at equal intervals in the horizontal direction and extending in the up-down direction, and first and second horizontal division lines disposed at equal intervals in the up-down direction, The processor adjusts a size of the third division composition map so that the target range and the inclusion range are located within the third division composition map and the exclusion range is located outside the third division composition map; The processor positions the third-division composition map so that the target range overlaps with an intersection of the first vertical division line and the first horizontal division line, and then trims the image along an outline of the third-division composition map to output the image.

Images