Program, method, information processing device, and system

Abstract

To provide a program, a method, an information processing device, and a system to improve convenience of editing moving images based on images captured by surveillance cameras or the like.SOLUTION: A program is designed to be executed by a computer with a processor and a memory. The program causes the processor to execute the steps of: acquiring captured moving images; analyzing the acquired moving images by using a plurality of analysis models, which is stored in advance, each of which is trained to detect a target object; presenting to a user a detection marker showing the timing at which the target object is detected by the analysis model in association with a timeline corresponding to the moving image; and generating the edited moving image from the captured moving image according to an instruction from the user based on the detection marker.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a program, a method, an information processing apparatus, and a system.

Background Art

[0002] Patent Document 1 describes a technique for specifying a tracking target before or during tracking of an object as a technique applicable to a surveillance camera system. In Patent Document 1, a user selects an object (tracking target) to be enlarged and displayed from among the extracted tracking target candidates to obtain a desired enlarged display image (zoom image).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, one type of tracking target is extracted in one frame. However, a plurality of types of tracking targets may be included in one frame. When editing a moving image based on an image captured by a surveillance camera or the like, it is required to improve convenience.

[0005] An object of the present disclosure is to improve convenience when editing a moving image based on an image captured by a surveillance camera or the like.

Means for Solving the Problems

[0006] A program for causing a computer including a processor and a memory to execute. The program causes the processor to perform steps of: acquiring a captured video; analyzing the acquired video using a plurality of pre-stored analysis models, each of which is trained to detect an object; presenting to a user a detection marker representing a timing at which an object is detected by the analysis model, in association with a timeline associated with the video; and generating an edited video from the captured video in response to an instruction from the user based on the detection marker.

Advantages of the Invention

[0007] According to the present disclosure, convenience can be improved when editing a video based on an image captured by a surveillance camera or the like.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiment for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0010] <Summary> The system according to this embodiment analyzes the acquired video with a plurality of image analysis models. The system presents an object that can grasp which model detected the target object, and creates an edited video based on the designation of the object.

[0011] <1 Configuration Diagram of the Whole System> FIG. 1 is a block diagram showing an example of the overall configuration of the system 1. The system 1 shown in FIG. 1 includes, for example, a terminal device 10, a server 20, a photographing device 31, a sensor 32, and a speaker 33. The terminal device 10, the server 20, and the speaker 33 are communicatively connected via, for example, a network 80.

[0012] In FIG. 1, an example in which the system 1 includes two terminal devices 10 is shown, but the number of terminal devices 10 included in the system 1 is not limited to two. The terminal device 10 is a terminal possessed by the user of the camera. The number of terminal devices 10 included in the system 1 may be less than three or may be three or more.

[0013] In this embodiment, an aggregate of a plurality of devices may be regarded as one server. The way of distributing the plurality of functions required to implement the server 20 according to this embodiment to one or a plurality of hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0014] The terminal device 10 shown in FIG. 1 may be, for example, a mobile terminal such as a smartphone or a tablet, or a stationary PC (Personal Computer) or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal.

[0015] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19.

[0016] The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with a device in the system 1 such as the server 20.

[0017] The input device 13 is a device for receiving input operations from the user (for example, a pointing device such as a touch panel, a touch pad, a mouse, etc., a keyboard, etc.).

[0018] The output device 14 is a device for presenting information to the user (a display, a speaker, etc.).

[0019] The memory 15 is for temporarily storing programs and data processed by programs, etc., and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0020] The storage 16 is for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive).

[0021] The processor 19 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0022] Server 20 is realized by, for example, a computer connected to network 80. As shown in FIG. 1, server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0023] Communication IF 22 is an interface for inputting and outputting signals so that server 20 can communicate with devices in system 1 such as terminal device 10.

[0024] Input / output IF 23 functions as an interface with an input device for receiving input operations from the user and an output device for presenting information to the user.

[0025] Memory 25 is for temporarily storing programs and data processed by programs and the like, and is a volatile memory such as DRAM, for example.

[0026] Storage 26 is for storing data, and is, for example, a flash memory or an HDD.

[0027] Processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0028] Imaging device 31 is a device that receives light by a light-receiving element and outputs it as an image signal. Imaging device 31 captures images at a frame rate that allows a series of movements to be recognized as a video. The frame rate at which a series of movements can be recognized as a video is, for example, about 30 fps. Imaging device 31 may be, for example, a camera capable of capturing a wide range of images of 360 degrees all around. In this case, imaging device 31 is realized by, for example, a camera with an ultra-wide-angle lens or a fish-eye lens.

[0029] The imaging device 31 is installed at a position where the entire scene can be overlooked without any obstacles at the scene. When one imaging device 31 cannot capture the entire scene, multiple imaging devices 31 are installed. When multiple imaging devices 31 are installed, for example, the section of the scene captured by one imaging device 31 is preset. The imaging device 31 outputs the acquired image signal to the server 20.

[0030] <1.1 Configuration of the terminal device> FIG. 2 is a block diagram showing a configuration example of the terminal device 10 shown in FIG. 1. The terminal device 10 shown in FIG. 2 is realized by a mobile terminal, a PC, or a wearable terminal. As shown in FIG. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 161, a position information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected by, for example, a bus or the like.

[0031] The communication unit 120 performs processes such as modulation / demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0032] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 is realized by, for example, a touch-sensitive device 131 where an instruction is input by touching the operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, a keyboard, a mouse, or the like. The input device 13 converts the instruction input by the user into an electrical signal and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a reception port for receiving an electrical signal input from an external input device.

[0033] The output device 14 is a device for presenting information to the user who operates the terminal device 10. The output device 14 is realized by, for example, a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized by, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like.

[0034] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts the signal given from the microphone 171 into a digital signal and gives the converted signal to the control unit 190. Also, the audio processing unit 17 gives the audio signal to the speaker 172. The audio processing unit 17 is realized by, for example, a processor for audio processing. The microphone 171 receives an audio input and gives an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal given from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0035] The camera 161 is a device that receives light by a light-receiving element and outputs it as an image signal.

[0036] The position information sensor 150 is a sensor that detects the position of the terminal device 10 and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals from at least three or four satellites are received, and based on the received signals, the current position of the terminal device 10 equipped with the GPS module is detected. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0037] The storage unit 180 is realized by, for example, a memory 15 and a storage 16 or the like, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0038] The user information 181 includes information about the user who uses the terminal device 10, for example. The information about the user includes, for example, information for identifying the user, the user's name, age, address, date of birth, phone number, email address, and the like.

[0039] The control unit 190 is realized by the processor 19 reading the program stored in the storage unit 180 and executing the instructions included in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0040] The operation reception unit 191 performs processing for receiving an instruction or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information based on an instruction input from the touch-sensitive device 131 or the like. The instruction input via the touch-sensitive device 131 or the like is, for example, an editing instruction from the user.

[0041] Also, the operation reception unit 191 receives a voice instruction input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. The operation reception unit 191 obtains an instruction from the user, for example, by analyzing the received voice signal and extracting a predetermined noun.

[0042] The transmission / reception unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 according to a communication protocol. Specifically, for example, the transmission / reception unit 192 transmits an editing instruction input from the user to the server 20. Also, the transmission / reception unit 192 receives information about the user from the server 20.

[0043] The presentation control unit 193 controls the output device 14 in order to present the information provided from the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the information transmitted from the server 20 to be displayed on the display 141. Further, the presentation control unit 193 causes the information transmitted from the server 20 to be output from the speaker 172.

[0044] <1.2 Functional Configuration of Server> FIG. 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0045] The communication unit 201 performs processing for the server 20 to communicate with an external device.

[0046] The storage unit 202 includes, for example, a photographing device database (DB) 2021, a detection log database (DB) 2022, and the like.

[0047] The photographing device DB 2021 is a database for holding information about cameras installed for photographing. Details will be described later.

[0048] The detection log DB 2022 is a database for holding information about objects detected by image analysis. Details will be described later.

[0049] The first learned model 2023 is a model generated by causing a machine learning model to perform machine learning according to a model learning program. The first learned model 2023 is, for example, a parameterized composite function in which a plurality of functions that perform predetermined inferences based on input data are combined. The parameterized composite function is defined by a combination of a plurality of adjustable functions and parameters. The learned model according to the present embodiment may be any parameterized composite function that satisfies the above requirements. The same applies to the second learned model 2024 and the third learned model 2025.

[0050] For example, when the first pre-trained model 2023 is generated using a feedforward multi-layer network, the parameterized composite function is defined as, for example, a combination of a linear relationship between layers using a weight matrix, a non-linear relationship (or linear relationship) using an activation function in each layer, and a bias. The weighted matrix and the bias are called the parameters of the multi-layer network. The parameterized composite function changes its form as a function depending on how the parameters are selected. In a multi-layer network, by appropriately setting the constituent parameters, a function capable of outputting a preferable result from the output layer can be defined. The same applies to the second pre-trained model 2024 and the third pre-trained model 2025.

[0051] As the multi-layer network according to this embodiment, for example, a deep neural network (DNN) which is a multi-layer neural network targeted for deep learning can be used. As the DNN, for example, a convolution neural network (CNN) targeted for images may be used.

[0052] The first pre-trained model 2023 is a model that learns to output information about a human based on the input image when the image is input. The first pre-trained model 2023 may be, for example, separate pre-trained models trained with separate training data respectively according to the information to be output.

[0053] For example, when an image is input, the first pre-trained model 2023 is trained to output information regarding human attributes such as gender, age, race, occupation, and the like. Specifically, in the present embodiment, the first pre-trained model 2023 identifies human attributes based on information about the physique, face, hairstyle, clothing, and items held by the human included in the image. Also, for example, the first pre-trained model 2023 may be trained to output information regarding a specific person when an image is input. Specifically, in the present embodiment, the first pre-trained model 2023 may identify a specific person based on information about the face, hairstyle, and physique of the specific person included in the image. Also, for example, the first pre-trained model 2023 may be trained to output information regarding human body parts when an image is input. Specifically, in the present embodiment, the first pre-trained model 2023 may identify human body parts such as hands or faces based on the skeletal arrangement of the human included in the image. At this time, the first pre-trained model 2023 uses a method such as OpenPose to identify the human body parts included in the image.

[0054] At this time, the training data uses, for example, images of humans as input data, and the judgments regarding the attributes, specific persons, and human body parts included in this input data are used as correct output data. In this case, the training data may not include the correct output data.

[0055] The second pre-trained model 2024 is a model trained to output information regarding an object based on the input image when an image is input. The object here refers to animals and plants, clothing, machines, vehicles, artworks, interior decorations, buildings, and the like. The second pre-trained model 2024 may be, for example, separate pre-trained models trained with separate training data according to the information to be output.

[0056] For example, when an image is input, the second pre-trained model 2024 is trained to output information regarding an object. Specifically, in the present embodiment, the second pre-trained model 2024 identifies the object based on information about the size, shape, material, color, accessories, and the like of the object included in the image.

[0057] At this time, for example, the learning data uses an image of an object as input data, and a judgment about the object represented by this input data is used as correct output data. In this case, the learning data may not include the correct output data.

[0058] The third trained model 2025 is a model that learns to output information related to operations based on the input image when an image is input. The third trained model 2025 may be, for example, separate trained models that are trained with separate learning data according to the information to be output.

[0059] For example, when an image is input, the third trained model 2025 learns to output information related to shopping. Specifically, in the present embodiment, when an image including, for example, the following content is input, the third trained model 2025 learns to output that a shopping-related action has been performed. · In an operation over a plurality of frames, a customer takes out money or a card near the cash register in a retail store · In an operation over a plurality of frames, a customer picks up a product in a retail store

[0060] At this time, for example, the learning data uses an image inside a retail store as input data, and a judgment about the operation represented by this input data is used as correct output data. In this case, the learning data may not include the correct output data.

[0061] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The control unit 203 functions as a reception control module 2031, a transmission control module 2032, an image analysis module 2034, an image editing module 2035, and a presentation module 2036 by operating according to the program.

[0062] The reception control module 2031 controls the process of the server 20 receiving a signal from an external device according to a communication protocol.

[0063] The transmission control module 2032 controls the process in which the server 20 transmits a signal to an external device according to a communication protocol.

[0064] The image analysis module 2034 analyzes the image captured by the imaging device 31 and outputs information about the objects at the scene. The image analysis module 2034 inputs the image to the first pre-trained model 2023, the second pre-trained model 2024, and the third pre-trained model 2025. More specifically, for example, the image analysis module 2034 analyzes the image using the first pre-trained model 2023, and after the analysis using the first pre-trained model 2023 is completed, analyzes the image using the second pre-trained model 2024. After the analysis using the second pre-trained model 2024 is completed, the image analysis module 2034 analyzes the image using the third pre-trained model 2025. The first pre-trained model 2023, the second pre-trained model 2024, and the third pre-trained model 2025 output information about the objects corresponding to each model according to the input image. The image analysis module 2034 outputs the information output from the first pre-trained model 2023, the second pre-trained model 2024, and the third pre-trained model 2025 to the presentation module 2036. The image analysis module 2034 does not necessarily have to use the first pre-trained model 2023, the second pre-trained model 2024, and the third pre-trained model 2025, and may analyze the image using a pre-trained model specified by the user. The image analysis module 2034 may input an image with a limited analysis range to the pre-trained model. The analysis range may be specified by the user, for example. Further, time may be added as a condition for analyzing the image. For example, the image analysis module 2034 analyzes the image only when an object continues to be detected for a certain period of time or longer within the analysis range.

[0065] The image editing module 2035 edits the video analyzed by the image analysis module 2034 in response to the editing operations by the user. Specifically, for example, the image editing module 2035 edits the video from viewpoints such as time, object, and range on the image. The image editing module 2035 may edit the video according to the user's individual operations, or may automatically edit the video based on the rules set by the user.

[0066] The presentation module 2036 presents information about the on-site object to the user via the terminal device 10. The presentation module 2036 may present information to the user based on an instruction from the user.

[0067] <2 Data Structure> Figures 4 to 5 are diagrams showing the data structure of the database stored in the server 20. Note that Figures 4 to 5 are examples and do not exclude data not described.

[0068] Figure 4 is a diagram showing the data structure of the imaging device DB2021. As shown in Figure 4, each record of the imaging device DB2021 includes, for example, the item "imaging device ID", the item "coordinates", the item "address", the item "model", the item "installation date", the item "removal date", the item "operation status", the item "latest analysis", the item "current analysis status", the item "analysis frequency", and the item "captured video".

[0069] The item "imaging device ID" indicates identification information for identifying the imaging device 31.

[0070] The item "coordinates" indicates the installation location of the imaging device 31. Specifically, the item "coordinates" indicates the latitude and longitude of the installation location of the imaging device 31. Note that the installation location of the imaging device 31 may be determined by indicators other than coordinates.

[0071] The item "address" indicates the address of the installation location of the imaging device 31.

[0072] The item "Model" indicates the model of the imaging device 31.

[0073] The item "Installation Date" indicates the date of installation of the imaging device 31.

[0074] The item "Removal Date" indicates the date of removal of the imaging device 31.

[0075] The item "Operation Status" indicates the operation status of the imaging device 31. Specifically, the item "Operation Status" is expressed as "Operating", "Stopped", "Faulty", or "Removed" according to the status of the imaging device 31.

[0076] The item "Latest Analysis" indicates the time point of the latest image among those analyzed by the server 20. Specifically, the server 20 analyzes the images received from the imaging device 31 at a predetermined timing, and the item "Latest Analysis" indicates the time point of the latest of those analyzed images. That is, it means that the images up to the time point represented by the item "Latest Analysis" in Figure 4 (example: 2023 / 10 / 22 23:59:59) have been completely analyzed.

[0077] The item "Current Analysis Status" indicates the time-based classification of the image currently being analyzed by the server 20. The time-based classification can be arbitrarily set. The classification is, for example, as follows. ·Daily (00:00:00~23:59:59) ·Morning (00:00:00~9:59:59), Afternoon (10:00:00~17:59:59), Evening (18:00:00~23:59:59)

[0078] The item "Analysis Frequency" indicates the frequency at which the server 20 analyzes images.

[0079] The item "Video" indicates the video captured by the imaging device 31. Specifically, the video is the video data captured by the imaging device 31. Also, the video may store reference information for a video data file located elsewhere.

[0080] FIG. 5 is a diagram showing the data structure of the detection log DB 2022. As shown in FIG. 5, each record of the detection log DB 2022 includes, for example, an item “imaging device ID”, an item “detection model”, an item “detection category”, an item “detection frame”, and an item “image area”. When an object is detected by a learned model, a new record is created in the detection log DB 2022, and each piece of information is stored in the corresponding item.

[0081] The item “imaging device ID” indicates identification information for identifying the imaging device 31.

[0082] The item “detection model” indicates the learned model that detected the object. Specifically, as the detection model, the first learned model 2023, the second learned model 2024, the third learned model 2025, etc. stored in the storage unit 202 are used.

[0083] The item “detection category” indicates the category of the detected object. Specifically, for example, the category is a classification of an object detected by a learned model. For example, an object detected by the first learned model 2023 is classified as “adult male” among humans.

[0084] The item “detection frame” indicates the frame of the image in which the object was detected.

[0085] The item “image area” indicates the area of the object on the image. Specifically, for example, after the image is divided into a mesh shape, the item “image area” indicates the mesh in which the object appears in the image.

[0086] <3 Operations> The operations of the server 20 that analyzes images will be described. Hereinafter, an example of a retail store will be used, but the application of the present embodiment is not limited to retail stores. For example, the present embodiment may be used for confirming rule implementation in a hospital or for safety management in a factory.

[0087] The imaging device 31 is attached, for example, to a ceiling from which the facilities inside the store can be overlooked. The facilities in the present embodiment are, for example, articles and locations used for purchase or sale. The facilities in the present embodiment include, for example, shelves, entrances and exits, cash registers, and shopping baskets. The imaging device 31 is imaging the store at a frame rate that enables recognition of a series of movements as a video. The frame rate that enables recognition of a series of movements as a video is, for example, about 30 fps. The imaging device 31 transmits the image data to the server 20.

[0088] The transmission frequency is a fixed cycle based on the data capacity of the untransmitted video. The imaging device 31 transmits the untransmitted video to the server 20, for example, every time the data capacity of the untransmitted video reaches a predetermined capacity. The transmission frequency may be a fixed cycle based on the time of the untransmitted video. The imaging device 31 transmits the untransmitted video to the server 20, for example, every time a predetermined time has elapsed since the latest transmission. Also, the imaging device 31 may immediately transmit the captured video to the server 20.

[0089] FIG. 6 is a flowchart showing the operation when recording the information detected by the image analysis in a log.

[0090] In step S61, the control unit 203 analyzes the image captured by the imaging device 31 using the first learned model 2023 recorded in the storage unit 202. Specifically, the image analysis module 2034 inputs, for example, the image data transmitted from the imaging device 31 into the first learned model 2023. The frequency at which the image analysis module 2034 analyzes the image is periodic. The image analysis module 2034 may analyze the image, for example, weekly, every few days, daily, or every few hours. The image analysis may be performed, for example, at 21:00 on every Sunday. Also, the image analysis module 2034 may analyze the image immediately after the server 20 receives the image. In other words, the frequency of image analysis by the image analysis module 2034 can be arbitrarily set.

[0091] The first learned model 2023 outputs information related to the detected object.

[0092] In step S62, the control unit 203 determines whether there is an object corresponding to the first learned model 2023 in the analyzed image. If there is a corresponding object, the control unit 203 causes the process to proceed to step S63. On the other hand, if there is no corresponding object, the control unit 203 causes the process to proceed to step S64.

[0093] In step S63, the control unit 203 records information related to the object detected in step S62 in the detection log DB 2022. The information to be recorded is, for example, the imaging device ID, the detection model, the detection category, the detection frame, and the image area.

[0094] In step S64, if there is a remaining learned model that has not been used for detection, the control unit 203 repeats the processes of steps S61, S62, and S63 using the unused learned model. If there is no remaining learned model that has not been used for detection, the control unit 203 ends the process. When the control unit 203 has ended the process, the control unit 203 may notify the display 141 of the terminal device 10 that the process has ended.

[0095] The user operates the terminal device 10 and accesses the server 20 after user authentication. The server 20 transmits information regarding the editing screen to the terminal device 10. The editing screen includes information of the detection log DB 2022 updated after the previous access. The user edits the video on the editing screen.

[0096] FIG. 7 is a schematic diagram showing an editing screen displayed on the display 141 of the terminal device 10. The display area 71 is an area for displaying a video in a streaming format. Marks 721, 722, and 723 are frame lines surrounding objects detected in the video. The mode of the frame line may be different for each learned model or detection category. The mode of the frame line includes, for example, color, thickness, shape, etc., or a combination thereof. Even when only an object of one detection model or only an object of one detection category is detected in the video, a mark may be attached to the object. The numerical value 73 indicates the time point of the displayed video. The icon group 74 is icons for operating the video. The icon group 74 is not limited to the three icons shown in FIG. 7.

[0097] The timeline 75 is a time axis for displaying the time associated with the detection marker 78 of each box 77 in time series. The timeline 75 is divided, for example, at regular time points. The playback marker 76 is a marker on the timeline 75 indicating the time point of the currently played video. The box 77 is a box indicating the category of the detected object. The box 77 may show information that can identify the learned model of the detected object. The detection marker 78 is a marker on the timeline 75 indicating the time when the object was detected. According to the example of FIG. 7, the time when the dog was detected is from 8:00 to 9:00.

[0098] Icon 79 is an icon for editing videos. For example, icon 79 is an icon for cutting out a video after dividing the video into fixed time intervals. For example, the user may determine the time of the video to be cut out by specifying the detection marker 78. That is, the image editing module 2035 cuts out the video in the time zone corresponding to the detection marker 78. The image editing module 2035 is not limited to matching the time zone of the video to be cut out with the detection marker 78, and a predetermined time width may be added to the detection marker 78. For example, the image editing module 2035 may add a time width of a predetermined length before the start point of the detection marker 78, after the end point of the detection marker 78, or both. Also, the user may determine the time of the video to be cut out by, for example, specifying the time width on the timeline 75. Specifically, for example, the image editing module 2035 receives from the user a tap or a drag at two points on the timeline 75, and cuts out the video of the received time width.

[0099] After editing the video, the image editing module 2035 saves the file of the edited video in, for example, the memory 25, or the storage 26, or the like.

[0100] FIG. 8 is a schematic diagram showing a part of an editing screen displayed on the display 141 of the terminal device 10. As shown in FIG. 8, the user may determine the time of the video to be cut out by, for example, specifying two or more detection markers. Specifically, as shown in the editing case 81, the user may determine the time of the video to be cut out by specifying two detection markers separated from each other on the timeline. That is, the image editing module 2035 may cut out the videos in the time zones corresponding to the respective detection markers specified by the user, and combine the respective videos. The two detection markers to be specified may belong to the same category or different categories.

[0101] Specifically, as shown in editing cases 82 and 83, the user may determine the time of the video to be clipped by specifying two detection markers that overlap on the timeline. In editing case 82, the image editing module 2035 extracts the overlapping part of the two detection markers specified by the user. In editing case 83, the image editing module 2035 extracts the part corresponding to at least one detection marker.

[0102] The icon 79 in FIG. 7 is not limited to the icon for clipping shown in FIG. 7. For example, the icon 79 may be an icon for processing a video. Specifically, the icon 79 may be, for example, an icon for adjusting the brightness of a video or an icon for applying a predetermined effect to a video. That is, the image editing module 2035 may receive an operation on the icon 79 from the user and process the video.

[0103] The user may arbitrarily set rules for editing a video. Specifically, the user may set rules for automatic editing based on conditions of an object. For example, the user may set a rule for automatically determining the time of the video to be clipped by specifying a detection marker 78 for a child and a detection marker 78 for a toy. Thereby, when a predetermined object is detected, a video including the object is automatically generated. Also, specifically, the user may set rules for automatic editing based on time conditions. For example, the user may set a rule for automatically increasing the brightness of a video taken at night. Thereby, a video is automatically generated at a predetermined time period. That is, the image editing module 2035 may automatically edit a video according to any video editing rules set by the user.

[0104] The icon 710 in FIG. 7 is an icon for downloading the video edited by the icon 79. The transmission control module 2032 transmits the edited video file to the terminal device 10 from the memory 25, the storage 26, etc. in response to an operation on the icon 710 by the user. The terminal device 10 stores the downloaded video file.

[0105] As described above, in the above embodiment, the control unit 203 acquires the captured video. The control unit 203 analyzes the video using a plurality of analysis models stored in advance, which are analysis models learned by the respective objects. The control unit 203 presents the detection marker 78 representing the timing when the object is detected by the analysis model to the user in association with the timeline 75 associated with the video. The control unit 203 generates an edited video from the captured video according to an instruction from the user based on the detection marker 78. Thereby, through the user interface in which the appearance time of the object for each model is visually arranged on the timeline 75, the user can intuitively and easily grasp the situation at the captured scene. Also, thereby, through the user interface for efficiently processing the object, the user can intuitively and easily edit the video.

[0106] Therefore, according to the system according to the present embodiment, convenience is improved when editing a video based on an image captured by a surveillance camera or the like.

[0107] Also, in the above embodiment, in the generating step, the control unit 203 accepts a designation for the detection marker 78 and generates an edited video of the time zone corresponding to the detection marker 78 or a time zone having a predetermined width in the time zone. Thereby, through the user interface for efficiently processing the object, the user can edit the video more intuitively and easily. Also, thereby, the user does not miss the sign before the object appears and the influence after the object exits.

[0108] Also, in the above embodiment, in the generating step, the control unit 203 receives a designation for a plurality of detection markers 78, and generates an edited video of a combined time zone obtained by connecting time zones corresponding to the detection markers 78, or a combined time zone obtained by connecting time zones having a predetermined width in the time zone. Thereby, through a user interface for efficiently processing an object, the user can edit the video more intuitively and simply. Further, thereby, the user does not miss signs before an object appears and the influence after the object exits the scene.

[0109] Also, in the above embodiment, in the generating step, when there is a time zone in which detection markers 78 by different analysis models overlap, the control unit 203 generates an edited video of the overlapping time zone, or a time zone having a predetermined width in the time zone. Thereby, through a user interface for efficiently processing an object, the user can edit the video more intuitively and simply. Further, thereby, the user does not miss signs before an object appears and the influence after the object exits the scene.

[0110] Also, in the above embodiment, in the generating step, when there is a time zone in which detection markers 78 by different analysis models overlap, the control unit 203 generates an edited video of a time zone corresponding to at least any one of the detection markers 78, or a time zone having a predetermined width in the time zone. Thereby, through a user interface for efficiently processing an object, the user can edit the video more intuitively and simply. Further, thereby, the user does not miss signs before an object appears and the influence after the object exits the scene.

[0111] In the above embodiment, in the generating step, the control unit 203 receives a specification of a time zone on the timeline 75 and generates an edited video of the specified time zone or a combined time zone obtained by connecting these time zones. Thereby, through a user interface for efficiently processing an object, the user can edit the video more intuitively and simply. Further, thereby, the user can edit the video while referring to the detection marker 78 without being restricted to the detection marker 78.

[0112] In the above embodiment, in the generating step, the control unit 203 receives, from the user, a rule for generating an edited video based on the detection marker 78 as an instruction. Thereby, through a user interface for efficiently processing an object, the user can edit the video more intuitively and simply. Further, thereby, the user can save the trouble of inputting frequently-occurring editing patterns.

[0113] In the above embodiment, in the analyzing step, the control unit 203 analyzes the video at an arbitrary frequency using an analysis model. Since the frequency of image analysis by the image analysis module 2034 can be arbitrarily set, the device according to the present embodiment is not necessarily required to have high performance.

[0114] In the above embodiment, in the presenting step, when objects detected by different analysis models are in the same frame, the control unit 203 simultaneously presents marks 721 to 723 representing the objects to the user. Thereby, even when a plurality of objects are detected in the same frame, the user can intuitively and simply grasp the situation at the photographed site based on the marks 721 to 723 attached to each object.

[0115] In the above embodiment, in the presenting step, the control unit 203 presents the marks 721 to 723 in different manners for each analysis model. Thereby, the user can intuitively and simply grasp the situation at the photographed site based on the marks 721 to 723 having different manners for each analysis model.

[0116] <4 Other Embodiments> In the above embodiment, it has been described that the imaging device 31 transmits a moving image to the server 20. However, the server 20 may be implemented by a general information processing device such as a PC. In this case, for example, by extracting the storage medium in which the moving image in the imaging device 31 is stored and connecting it to the input port of the information processing device, the information stored in the storage medium is input to the information processing device. The control unit of the information processing device performs the same processing as the image analysis module 2034 of the server 20. For example, it analyzes the image captured by the imaging device 31 and outputs information regarding the objects at the scene. Further, the control unit of the information processing device performs the same processing as the image editing module 2035 of the server 20. For example, it receives an editing instruction from the user and generates an edited moving image based on the received editing instruction.

[0117] Also, in the above embodiment, it has been described that the edited moving image is downloaded from the memory 25, or the storage 26, etc. to the terminal device 10 and saved. However, the user may view the moving image edited on the server 20 without downloading it.

[0118] Also, in the above embodiment, it has been described that the image analysis module 2034 may limit the analysis range of the image. The number of analysis ranges in one image may be two or more. For example, the analysis range in the captured moving image of a retail store may be two locations, the entrance and the exit. Also, when an object is detected simultaneously in two or more analysis ranges, or when an object is detected with a difference within a certain time period, the moving image of the detected time may be extracted.

[0119] Also, in the above embodiment, it has been described that a moving image of a time period obtained by adding a predetermined time width to the detection marker may be cut out. However, the created moving image is not limited to this. The image editing module 2035 may cut out a moving image of a time period consisting only of the predetermined time width.

[0120] Also, in the above embodiment, the case where one video captured by the one imaging device 31 and one timeline corresponding to the video are displayed on the editing screen has been described. However, the content displayed on the editing screen is not limited to these. That is, when generating an edited video, the videos to be referenced are not limited to those captured by the one imaging device 31. For example, the image analysis module 2034 may analyze videos captured by two or more imaging devices 31. Also, the image editing module 2035 may generate an edited video based on videos captured by two or more imaging devices 31. Specifically, for example, the image analysis module 2034 analyzes a plurality of videos captured by a plurality of imaging devices 31 installed to capture the same scene from different angles. The image editing module 2035 generates an edited video based on the analysis result.

[0121] FIG. 9 is a schematic diagram showing another example of the editing screen displayed on the display 141 of the terminal device 10. Note that in FIG. 9, the case where one object is detected is shown, but a plurality of objects may be detected. That is, a plurality of learned models may be used for analysis. The editing screen in FIG. 9 includes, for example, a plurality of videos and one timeline that collectively corresponds to those videos.

[0122] In FIG. 9, the detection marker 78 indicates that an object has been detected in the video. As shown by the detection marker 78 in FIG. 9, the presentation module 2036 may represent a detection marker for each video of each camera under the same object. By operating a plurality of detection markers under the same object, the user can extract videos, for example, based on the time when the same object has been commonly detected in a plurality of videos.

[0123] As shown by the display area 71 in FIG. 9, the plurality of videos are displayed in divided display areas within the editing screen.

[0124] The image editing module 2035 may, for example, provide the user with options to select videos to be edited. Specifically, for example, the image editing module 2035 may be configured to receive from the user a selection of videos included in the video to be edited. When one or more detection markers under the same object are selected by the user, the image editing module 2035, for example, targets the video corresponding to the selected detection marker for editing. The user may select a video in which no detection marker appears under the same object.

[0125] When the user selects a single video, the image editing module 2035 edits the selected video in the same way as when one video and one timeline corresponding to that video are displayed on the editing screen.

[0126] When the user selects multiple videos, the image editing module 2035 defines a method for playing the multiple videos according to the user's specification and then edits the selected videos. That is, the user specifies whether the selected multiple videos are to be played sequentially or in parallel.

[0127] When multiple videos are played sequentially, the order of playback is determined automatically or by an operation from the user. The order of playback determined automatically may be determined, for example, from the perspectives of time, the proportion occupied on the screen, etc. Specifically, the image editing module 2035 determines the order of playback, for example, considering the length of time indicated by the detection markers of the videos, the average proportion of the area of the object occupying the screen throughout the video, etc. Also, when the user determines the order of playback, the image editing module 2035 may propose to the user the order determined automatically as an aid to the determination.

[0128] When a plurality of videos are played in parallel, the plurality of videos are played, for example, in a picture-in-picture mode in which a subordinate video is arranged above the main video in a video playback area. The primary-secondary relationship of the videos is determined automatically or by an operation from the user. The primary-secondary relationship determined automatically may be determined from viewpoints such as time, the ratio occupied on the screen, etc. Specifically, the image editing module 2035 determines the primary-secondary relationship in consideration of, for example, the length of time indicated by the detection marker of the video, the average ratio of the area of the object occupying the screen throughout the video, etc. Also, when the user determines the primary-secondary relationship, the image editing module 2035 may propose to the user the primary-secondary relationship determined automatically as an aid for the determination.

[0129] Also, when a plurality of videos are played in parallel, the plurality of videos may be played, for example, in a divided video playback area. That is, the image editing module 2035 divides the video playback area so as to play a plurality of videos on the video playback area. The division of the area is determined automatically or by an operation from the user. The division determined automatically may be determined from viewpoints such as time, the ratio occupied on the screen, etc. Specifically, the image editing module 2035 determines the division in consideration of, for example, the length of time indicated by the detection marker of the video, the average ratio of the area of the object occupying the screen throughout the video, etc. Also, when the user determines the division, the image editing module 2035 may propose to the user the division determined automatically as an aid for the determination. Even if there are differences among the plurality of videos from viewpoints such as time, the ratio occupied on the screen, etc., the image editing module 2035 may evenly divide the video playback area by an operation from the user.

[0130] Also, in the above embodiment, the case where the detection marker regarding the detected object appears on the timeline has been described. However, information other than the detected object may appear on the timeline. For example, the presentation module 2036 may represent the output information of a predetermined IoT device on a timeline. Specifically, the reception control module 2031 receives, at a predetermined period, information measured by an IoT device from the IoT device attached to the on-site equipment. The presentation module 2036 presents a detection marker based on the information transmitted from the IoT device on the timeline of the editing screen. The presentation module 2036 presents a detection marker on the timeline of the editing screen, for example, when the acquired information satisfies a predetermined requirement. For example, when the reception control module 2031 receives information regarding a temperature equal to or higher than a certain temperature from a temperature sensor attached to the equipment shown in the image, the presentation module 2036 presents a detection marker indicating the time at which the temperature is reached on the timeline. The presentation module 2036 may present the detection marker such that the detection marker indicates the temperature range in a different manner for each temperature range. For example, an orange detection marker may indicate a temperature of 30 degrees Celsius or higher and less than 40 degrees Celsius. Also, for example, when the reception control module 2031 receives information regarding an acceleration equal to or higher than a certain acceleration from an acceleration sensor attached to the equipment shown in the image, the presentation module 2036 presents a detection marker indicating the time at which the acceleration is reached on the timeline. The presentation module 2036 may present the detection marker such that the detection marker indicates the acceleration range in a different manner for each acceleration range. For example, a blue detection marker may indicate an acceleration of 1G or higher and less than 2G.

[0131] <Basic Hardware Configuration of a Computer> FIG. 10 is a block diagram showing the basic hardware configuration of a computer 100. The computer 100 includes at least a processor 101, a main memory device 102, an auxiliary storage device 103, and a communication IF 109 (interface). These are electrically connected to each other by a bus.

[0132] The processor 101 is hardware for executing an instruction set described in a program. The processor 101 is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0133] The main memory device 102 is for temporarily storing programs and data processed by programs and the like. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0134] The auxiliary storage device 103 is a storage device for storing data and programs. For example, it includes flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, and the like.

[0135] The communication IF 109 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards.

[0136] The network is composed of various mobile communication systems and the like constructed by the Internet, LAN, wireless base stations, and the like. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks connectable to the Internet by a predetermined access point (for example, Wi-Fi (registered trademark)), and the like. When connecting wirelessly, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), and the like. When connecting wired, the network includes those directly connected by a USB (Universal Serial Bus) cable or the like.

[0137] Note that all or part of each hardware configuration can be provided in a distributed manner across multiple computers 100 and connected to each other via a network, thereby virtually realizing the computer 100. In this way, the computer 100 is a concept that includes not only a single housing or a computer 100 housed in a case, but also a virtualized computer system.

[0138] <Basic Functional Configuration of Computer 100> The functional configuration of the computer realized by the basic hardware configuration of the computer 100 shown in FIG. 10 will be described. The computer includes at least functional units of a control unit, a storage unit, and a communication unit.

[0139] Note that the functional units included in the computer 100 can also be realized by distributing all or part of each functional unit across multiple computers 100 interconnected by a network. The computer 100 is a concept that includes not only a single computer 100, but also a virtualized computer system.

[0140] The control unit is realized by the processor 101 reading out various programs stored in the auxiliary storage device 103 and expanding them in the main storage device 102, and executing processing according to the programs. The control unit can realize a functional unit that performs various information processes according to the type of program. Thereby, the computer is realized as an information processing device that performs information processing.

[0141] The storage unit is realized by the main storage device 102 and the auxiliary storage device 103. The storage unit stores data, various programs, and various databases. Also, the processor 101 can secure a storage area corresponding to the storage unit in the main storage device 102 or the auxiliary storage device 103 according to a program. Also, the control unit can cause the processor 101 to execute processes of adding, updating, and deleting data stored in the storage unit according to various programs.

[0142] The database refers to a relational database, which is for managing a set of data called a tabular table structurally defined by rows and columns by associating them with each other. In a database, a table is called a table, a column of a table is called a column, and a row of a table is called a record. In a relational database, the relationships between tables can be set and associated. Normally, each table is set with a column that serves as a key for uniquely identifying records, but setting a key for a column is not essential. The control unit can cause the processor 101 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

[0143] The communication unit is realized by the communication IF 109. The communication unit realizes the function of communicating with other computers 100 via a network. The communication unit can receive information transmitted from other computers 100 and input it to the control unit. The control unit can cause the processor 101 to execute information processing on the received information according to various programs. Also, the communication unit can transmit the information output from the control unit to other computers 100.

[0144] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry including a general-purpose processor, a specific-purpose processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), a conventional circuit, and / or a combination thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is regarded as circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory. As used herein, circuitry, units, and means are hardware programmed to perform the described functions or hardware that performs them. The hardware may be any hardware disclosed herein or any hardware known to be programmed or perform to perform the described functions. If the hardware is a processor considered to be of the circuitry type, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or the processor.

[0145] Some embodiments of the present disclosure have been described above. However, these embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are to be included in the scope and gist of the invention, as well as in the invention described in the claims and the equivalent scope thereof.

[0146] <Appendix> The matters described in each of the above embodiments are appended below. (Appendix 1) A program for operating a computer including a processor 29 and a memory 25, the program causing the processor to perform steps of: acquiring a captured video; analyzing the acquired video using a plurality of pre-stored analysis models each learned to detect an object; presenting to a user a detection marker 78 representing a timing at which an object is detected by the analysis model in association with a timeline 75 associated with the video; and generating an edited video from the captured video in response to an instruction from the user based on the detection marker. (Appendix 2) In the generating step, upon receiving a specification for a detection marker, a program according to (Appendix 1) that generates an edited video of a time zone corresponding to the detection marker or a time zone having a predetermined width in the time zone. (Appendix 3) In the generating step, upon receiving a specification for a plurality of detection markers, a program according to (Appendix 1) or (Appendix 2) that generates an edited video of a combined time zone obtained by connecting time zones corresponding to the detection markers or a combined time zone obtained by connecting time zones having a predetermined width in the time zone. (Appendix 4) In the generating step, when there is a time zone where detection markers by different analysis models overlap, a program according to any one of (Appendix 1) to (Appendix 3) that generates an edited video of the overlapping time zone or a time zone having a predetermined width in the time zone. (Appendix 5) In the generating step, when there is a time zone where detection markers by different analysis models overlap, a program according to any one of (Appendix 1) to (Appendix 4) that generates an edited video of a time zone corresponding to at least any one of the detection markers or a time zone having a predetermined width in the time zone. (Appendix 6) In the generating step, upon receiving a specification of a time zone on a timeline, a program according to any one of (Appendix 1) to (Appendix 5) that generates an edited video of the specified time zone or a combined time zone obtained by connecting these time zones. (Appendix 7) In the generating step, a program according to any one of (Appendix 1) to (Appendix 6), in which a rule for generating an edited video based on a detection marker is given as an instruction from a user. (Appendix 8) In the analyzing step, a program according to any one of (Appendix 1) to (Appendix 7) that analyzes a video at an arbitrary frequency using an analysis model. (Appendix 9) In the step of prompting, when objects detected by different analysis models are in the same frame, a program according to any one of (Appendix 1) to (Appendix 8) that simultaneously prompts the user with marks representing the objects. (Appendix 10) In the step of prompting, a program according to (Appendix 9), where the marks are in different forms for each analysis model. (Appendix 11) A method executed by a computer including a processor and a memory, wherein the processor executes all steps executed in the invention according to any one of (Appendix 1) to (Appendix 10). (Appendix 12) An information processing apparatus including a control unit 203 and a storage unit 202, wherein the control unit executes all steps executed in the invention according to any one of (Appendix 1) to (Appendix 10). (Appendix 13) A system including means for executing all steps executed in the invention according to any one of (Appendix 1) to (Appendix 10).

Explanation of Reference Numerals

[0147] 1... System 10... Terminal device 12... Communication IF 120... Communication unit 13... Input device 131... Touch-sensitive device 14... Output device 141... Display 15... Memory 150... Position information sensor 16... Storage 160... Camera 17... Audio processing unit 171... Microphone 172... Speaker 180... Storage unit 181... User information 19... Processor 190... Control unit 191... Operation reception unit 192… Transmission and reception unit 193… Presentation control unit 20… Server 201… Communication unit 202… Memory unit 2021… Imaging device database 2022… Detection log database 2023… First trained model 2024… Second trained model 2025… Third trained model 203… Control unit 2031… Reception control module 2032… Transmission control module 2034… Image analysis module 2035… Image editing module 2036… Presentation module 22… Communication interface 23… Input / output interface 25… Memory 26… Storage 29… Processor 31… Imaging device 80… Network 100… Server (computer) 101… Processor 102… Memory (main memory device) 103… Storage (auxiliary storage device) 109… Communication interface

Claims

1. A program for operating a computer including a processor and a memory, the program causing the processor to acquire a captured video; analyze the acquired video using a plurality of pre-stored analysis models each trained to detect an object; present to a user a detection marker indicating a timing at which an object is detected by the analysis model, in association with a timeline associated with the video; generate an edited video from the captured video according to an instruction from the user based on the detection marker and execute the program

2. The program according to claim 1, wherein in the generating step, a designation for a detection marker is received, and an edited video of a time period corresponding to the detection marker or a time period having a predetermined width in the time period is generated.

3. The program according to claim 1, wherein in the generating step, designations for a plurality of detection markers are received, and an edited video of a combined time period obtained by connecting time periods corresponding to the detection markers or a combined time period obtained by connecting time periods having a predetermined width in the time period is generated.

4. The program according to claim 1, wherein in the generating step, when there is a time period in which detection markers by different analysis models overlap, an edited video of the overlapping time period or a time period having a predetermined width in the time period is generated.

5. The program according to claim 1, wherein in the generating step, when there is a time period in which detection markers by different analysis models overlap, an edited video of at least a time period corresponding to any one of the detection markers or a time period having a predetermined width in the time period is generated.

6. The program according to claim 1, wherein in the generating step, a designation of a time period on a timeline is received, and an edited video of the designated time period or a combined time period obtained by connecting these time periods is generated.

7. The program according to claim 1, wherein in the generating step, a rule for generating the edited video based on the detection marker is given as the instruction from the user.

8. The program according to claim 1, wherein in the step of analyzing, the video is analyzed at an arbitrary frequency using the analysis model.

9. The program according to claim 1, wherein in the step of presenting, when objects detected by different analysis models are in the same frame, marks representing the objects are presented to the user simultaneously.

10. The program according to claim 9, wherein in the step of presenting, the marks are in different forms for each analysis model.

11. A method executed by a computer comprising a processor and a memory, wherein the processor executes all the steps executed in the invention according to any one of claims 1 to 10.

12. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit executes all the steps executed in the invention according to any one of claims 1 to 10.

13. A system comprising means for executing all the steps executed in the invention according to any one of claims 1 to 10.

Images