Album automatic creation system

Abstract

To provide an album automatic creation system that allows a user to select images to be posted on SNS etc., by automatically extracting unnecessary images and excluding the unnecessary images from an album in advance.SOLUTION: An album automatic creation system 100 comprises a photography system 1 which photographs moving images and a server 2 which extracts vehicle images as images showing a vehicle from the moving image, and further extracts unnecessary images considered to be unnecessary as images to be posted on social networking service from the vehicle images. The server 2 creates an album using vehicle images excluding the unnecessary image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an album automatic creation system.

Background Art

[0002] Users who like driving may have a desire (need) to take pictures of the appearance of their own vehicle while driving. By posting the taken images on, for example, a social networking service (hereinafter referred to as "SNS"), the images can be seen by many people. Since it is difficult for a user to take pictures of the appearance of a vehicle while driving by himself / herself, services for taking pictures of the appearance of a vehicle while driving have been proposed. For example, Japanese Unexamined Patent Application Publication No. 2021-190909 discloses an album automatic creation system that extracts a plurality of frames including a specific vehicle from video data, generates an image (vehicle image) including the vehicle, and creates an album using the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The user selects an image for posting to SNS or the like from the images stored in the album (hereinafter referred to as "appreciation images"). Here, the albums created by the conventional automatic album creation system include images that are unnecessary for use on SNS, such as images that cause discomfort to viewers when posted on SNS (hereinafter referred to as "unnecessary images"). Specifically, they are images in which inappropriate behaviors such as getting out of the vehicle or speeding are displayed. In addition, there are images in which personal information is specified (images in which the faces of the driver or passengers can be identified), etc. The unnecessary images stored in the album hinder the user's work of selecting an image for posting from the album to SNS.

[0005] The present disclosure has been made to solve the above problems, and its purpose is to provide an automatic album creation system that automatically extracts unnecessary images and excludes the unnecessary images from the album in advance, thereby facilitating the user's selection of images to be posted to SNS or the like.

Means for Solving the Problems

[0006] The automatic album creation system of the present disclosure includes a shooting system for shooting a video, and a server that extracts vehicle images, which are images in which a vehicle is displayed, from the video, and extracts unnecessary images, which are images unnecessary as images to be posted to SNS, from the vehicle images. The server creates an album using the vehicle images excluding the unnecessary images.

[0007] According to the above configuration, the server can create an album from which unnecessary images are excluded by extracting unnecessary images from the vehicle images. Thereby, it facilitates the user's work of selecting an image to be posted from the album to SNS.

Effects of the Invention

[0008] According to the present disclosure, it is to provide an automatic album creation system that facilitates the user's work of selecting an image to be posted from the album to SNS.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated.

[0011] <System Configuration> FIG. 1 is a diagram showing an album automatic creation system according to the present embodiment. The album automatic creation system 100 includes a plurality of photographing systems 1 and a server 2. Each of the plurality of photographing systems 1 and the server 2 are communicably connected to each other via a network NW. Although three photographing systems 1 are shown in FIG. 1, the number of photographing systems 1 is not particularly limited. There may be only one photographing system 1.

[0012] The photographing system 1 includes a camera (not shown), a memory, a processor, and a communication IF. The photographing system 1 is installed, for example, near a road. The photographing system 1 photographs a vehicle traveling on the road with the camera and stores the photographed video (including a plurality of temporally continuous photographs) in the memory. The photographing system 1 performs a predetermined arithmetic process described later on the video by the processor. The photographing system 1 transmits the result of the arithmetic process together with the video to the server 2 using the communication IF.

[0013] Server 2 includes a processor (not shown), a memory, and a communication IF. Server 2 is, for example, the in-house server of an operator that provides a vehicle photography service. Server 2 receives a video and its calculation processing result from the photography system 1 via the communication IF. Server 2 performs predetermined calculation processing (described later) on the received video using the processor, and further performs image processing to generate an image (hereinafter referred to as a "viewing image") for the user to view. Further, Server 2 creates an album using the viewing image by means of the processor. Server 2 stores the program executed by the processor and the data used in the program in the memory. Also, Server 2 stores the data used for image processing in the memory, or stores the viewing image in the memory. The generated viewing image is provided to the user through the album. The viewing image is generally a still image, but may be a video of a specified time (for example, a short time of about several seconds).

[0014] <Functional Configuration of Image Processing System> FIG. 2 is a functional block diagram showing the functional configuration of the photography system and the server according to the present embodiment. The photography system 1 includes a photography unit 31, a communication unit 32, and a calculation processing unit 33.

[0015] The photography unit 31 photographs a video of a vehicle and outputs the photographed video to a memory (not shown) of the photography system 1.

[0016] The communication unit 32 performs two-way communication with the communication unit 42 (described later) of the server 2 via the network NW. The communication unit 32 transmits a video (more specifically, a video cut out so as to include the vehicle of the target vehicle specifying unit 431) to the server 2. Here, the target vehicle is a vehicle that a user who receives the vehicle photography service wishes to photograph.

[0017] The arithmetic processing unit 33 extracts vehicles (not limited to the target vehicle, but all vehicles in general) from the video captured by the imaging unit 31 by known vehicle extraction processing using machine learning techniques such as deep learning. After that, the arithmetic processing unit 33 selects the target vehicle from the extracted vehicles by known target vehicle selection processing.

[0018] The arithmetic processing unit 33 extracts the feature amounts of the target vehicle by analyzing the video including the target vehicle. The feature amounts are the driving state and appearance of the target vehicle, etc. More specifically, the arithmetic processing unit calculates the driving speed of the target vehicle based on the temporal changes of the target vehicle in the frame including the target vehicle (for example, the movement amount of the target vehicle between frames, the change amount of the size of the target vehicle between frames). In addition to the driving speed of the target vehicle, the arithmetic processing unit may calculate, for example, the acceleration (deceleration) of the target vehicle. Also, the arithmetic processing unit extracts information regarding the appearance (body shape, body color, etc.) of the target vehicle using known image recognition techniques. The arithmetic processing unit transmits the feature amounts (driving state and appearance) of the target vehicle to the server 2. The arithmetic processing unit 33 associates the extracted vehicle and feature amounts with the identifier of the frame for each frame of the video. The identifier of the frame is, for example, a time stamp (time information of the frame).

[0019] Also, the arithmetic processing unit cuts out all the frames including the target vehicle from the video stored in the arithmetic processing unit. The cut-out video is output to the communication unit 32.

[0020] The server 2 includes a storage unit 41, a communication unit 42, and an arithmetic processing unit 43. The storage unit 41 includes an image storage unit 411 and a registration information storage unit 412. The arithmetic processing unit 43 includes a target vehicle identification unit 431, an image processing unit 434, an unnecessary image extraction unit 433, an album creation unit 435, and a web service management unit 436.

[0021] The image storage unit 411 stores the appreciation images obtained as a result of the arithmetic processing by the server 2. More specifically, the image storage unit 411 stores the image selected by the vehicle image extraction unit 432 (hereinafter referred to as "vehicle image"), the image extracted by the unnecessary image extraction unit 433 (hereinafter referred to as "unnecessary image"), the image processed by the image processing unit 434 (appreciation image), and also stores the album created by the album creation unit 435.

[0022] The registration information storage unit 412 stores the registration information related to the vehicle photography service. The registration information includes the personal information of the user who has applied for the provision of the vehicle photography service and the vehicle information of that user. The personal information of the user includes information regarding, for example, the user's identification number (ID), name, date of birth, address, telephone number, email address, etc. The vehicle information of the user includes information regarding the number of the vehicle number plate. The vehicle information may include information regarding, for example, the vehicle type, model year, body shape (sedan type, wagon type, one-box type, etc.), body color, etc.

[0023] The communication unit 42 performs two-way communication with the communication unit 32 of the photography system 1 via the network NW. The communication unit 42 transmits the number of the target vehicle to the photography system 1 or receives the number of each vehicle photographed by the photography system 1. Further, the communication unit 42 receives the video including the target vehicle and the feature amounts (driving state and appearance) of the target vehicle from the photography system 1.

[0024] The target vehicle identification unit 431 extracts the frame or image in which a vehicle (not limited to the target vehicle, but vehicles in general) is displayed by a known vehicle extraction process using a machine learning technique such as deep learning from the video received from the photography system 1. Further, in each of the frames or images extracted by the known target vehicle identification process, the target vehicle is identified. The target vehicle identification unit 431 outputs the frame or image including the identified target vehicle to the vehicle image extraction unit 432.

[0025] The vehicle image extraction unit 432 selects at least one image from the frame or image including the specified target vehicle input from the target vehicle identification unit 431. The selected image may be selected for all the frames input from the target vehicle identification unit 431, or only the images that meet the preset conditions may be selected. The conditions are, for example, whether the whole body of the target vehicle is included, or whether the target vehicle occupies a certain proportion or more of the picture angle. The vehicle image extraction unit 432 outputs the vehicle image to the unnecessary image extraction unit 433. Note that the vehicle image is generally a still image, but may also be a moving image within a specified time (for example, a short time of about several seconds).

[0026] The unnecessary image extraction unit 433 extracts images (unnecessary images) that are unnecessary as images to be posted on SNS. More specifically, it extracts images in which inappropriate behaviors such as the driver or the like sticking their body out of the window are displayed from the vehicle images. The process for extracting unnecessary images is called "unnecessary image extraction process". For the unnecessary image extraction process, a learned model generated by a machine learning technique such as deep learning can also be used. In this example, the unnecessary image extraction unit 433 is realized by the "unnecessary image extraction model" described later. The unnecessary image extraction unit 433 outputs the vehicle image excluding the unnecessary images to the image processing unit 434. Note that the unnecessary image extraction process may be performed by a technique with a known configuration.

[0027] The image processing unit 434 performs various image corrections (trimming, color correction, distortion correction, etc.) on the images input from the unnecessary image extraction unit 433. The image processing unit 434 outputs the processed appreciation image to the album creation unit 435. The image processing unit 434 may also output only the images that look good in the photos from among the plurality of images to the album creation unit 435.

[0028] The album creation unit 435 creates an album using the appreciation images input from the image processing unit 434. Known image analysis technologies (for example, technologies for automatically creating photo books, slide shows, etc. from images taken with a smartphone) can be used for album creation. The album creation unit 435 outputs the album to the web service management unit 436.

[0029] The web service management unit 436 provides a web service (for example, an application program that can be linked to an SNS) using the album created by the album creation unit 435. Note that the web service management unit 436 may be implemented on a server different from server 2.

[0030] <Trained model> An example of a trained model (unnecessary image extraction model) used for the unnecessary image extraction process will be described. The unnecessary image extraction model takes a vehicle image as input and outputs an unnecessary image. The estimation model, which is the pre-trained model, includes, for example, a neural network and parameters. The neural network is a known neural network used for image recognition processing by deep learning. The estimation model is trained by a known learning method using example data and correct answer data, and the trained model after learning is stored in the unnecessary image extraction unit 433. A large number of teacher data including example data and correct answer data are prepared in advance by the developer. Specifically, the example data is image data including an image in which an inappropriate act (such as an act of a driver leaning out of the window) to be extracted is displayed. The correct answer data is an image in which the inappropriate act included in the example data is displayed.

[0031] <Processing flow> Figure 3 is a flowchart showing the processing procedure of the album automatic creation system in the embodiment of the present disclosure. This flowchart is executed, for example, when a predetermined condition is satisfied or at every predetermined cycle.

[0032] In step S10, the imaging system 1 captures a video including the target vehicle. In step S15, the imaging system 1 extracts the vehicles and the target vehicle from the video, and extracts the feature amounts of the target vehicle. The imaging system 1 transmits the video cut out to include the target vehicle and the feature amounts of the target vehicle to the server 2.

[0033] In step S20, the server 2 extracts the frames in which the vehicle is displayed from the video received from the imaging system 1. The server 2 identifies the target vehicle based on the feature amounts of the target vehicle from among the extracted frames in which the vehicle is displayed.

[0034] In step S25, the server 2 selects at least one image (vehicle image) from among the frames including the target vehicle.

[0035] In step S30, the server 2 extracts the images (unnecessary images) that are unnecessary as the images to be posted on the SNS from among the vehicle images.

[0036] In step S35, the server 2 performs processing on the vehicle images excluding the unnecessary images to generate appreciation images.

[0037] In step S40, the server 2 creates an album using the appreciation images. The user can view the created album or post the desired images in the album on the SNS.

[0038] As described above, in the present embodiment, the server 2 extracts the unnecessary images that are unnecessary as the images to be posted on the SNS from among the created vehicle images, performs processing on the vehicle images excluding the unnecessary images, and generates appreciation images. The server 2 creates an album using the appreciation images. Thereby, the server 2 can create an album that does not include unnecessary images. As a result, the album automatic creation system 100 can facilitate the work of the user to select the images to be posted on the SNS from within the album.

[0039] Since the server 2 identifies the target vehicle from among a plurality of vehicles displayed in the vehicle image, it is not affected by vehicles other than the target vehicle in the inappropriate image extraction process.

[0040] In addition, in the present embodiment, an example in which the imaging system 1 and the server 2 share and execute image processing has been described. However, the imaging system 1 may execute all image processing, transmit the processed image data (appreciation image) to the server 2, and the arithmetic processing unit 43 of the server 2 may have only the functions of the album creation unit 435 and the web service management unit 436.

[0041] In the above embodiment, as a specific example of the unnecessary image extracted by the unnecessary image extraction unit 433, an image in which an inappropriate act (such as an act of a driver or the like leaning out of the window) is displayed has been shown as an example, but the present disclosure is not limited to this.

[0042] For example, first, the unnecessary image may include an image in which information such that personal information is specified is displayed (hereinafter referred to as a "personal information specifying image"). Specifically, the personal information is the face of the driver or a passenger. The process in which the unnecessary image extraction unit 433 extracts a personal information specifying image from the vehicle image is referred to as a "personal information specifying image extraction process". For the personal information specifying image extraction process, a learned model generated by a machine learning technique such as deep learning can also be used. This is referred to as a personal information specifying image extraction model. The example data of this learned model is image data including an image in which the face of a driver or the like to be extracted is displayed. The correct data is an image in which the face of a person such as a driver included in the example data is displayed. The personal information specifying image extraction model takes a vehicle image as an input and outputs a personal information specifying image. Note that the information such that personal information is specified is not limited to the face of an individual. Note that the personal information specifying image extraction process may be performed by a technique having a known configuration.

[0043] Second, the unnecessary image extraction unit 433 may define an image in which speeding driving is displayed (including a video of a specified time) (hereinafter referred to as a "hazardous driving specific image") as an unnecessary image. The hazardous driving specific image may be extracted from a vehicle image based on a feature amount related to the driving state of the target vehicle, or may be extracted by a technique based on a known configuration.

[0044] The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The technical scope shown by the present disclosure is indicated by the claims rather than the description of the above-described embodiments, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Reference Numerals

[0045] 1 Imaging system, 2 Server, 31 Imaging unit, 32, 42 Communication unit, 33, 43 Arithmetic processing unit, 41 Storage unit, 100 Image processing system, 411 Image storage unit, 412 Registered information storage unit, 431 Target vehicle identification unit, 432 Vehicle image extraction unit, 433 Unnecessary image extraction unit, 434 Image processing unit, 435 Album creation unit, 436 Web service management unit.

Claims

Claim 1 A photographing system for photographing videos, a server that extracts a vehicle image, which is an image in which a vehicle is displayed, from the video, extracts an unnecessary image, which is an image that is unnecessary as an image to be posted on a social networking service, from the vehicle image, and creates an album using the vehicle image excluding the unnecessary image, An automatic album creation system comprising the above.

Images