Image processing method, computer program, and image processing device

Abstract

Provided are an image processing method, a computer program, and an image processing device with which it is possible to expect to detect a change in a scene from a moving image with high accuracy. In an image processing method of the present embodiment, an image processing device detects candidates of a scene change position from a moving image, detects candidate frames that can be a main frame of a scene from among frames constituting the moving image, determines the main frame from among the candidate frames, and transmits the determined main frame to the moving image. And determining the scene change position from the candidates of the scene change position based on the candidates of the scene change position and the sequential order of the main frames. The scene change position can be determined by removing any one of the two scene change positions from the candidates in the case where there is no main frame between the two scene change position candidates arranged in time sequence.

Description

Technical Field

[0001] The present invention relates to an image processing method, a computer program, and an image processing device for performing image processing on a dynamic image. Background Art

[0002] Image processing techniques for detecting changes in scenes (scenes or shots, etc.) in dynamic images are widely used. In such techniques, for example, for a plurality of frames constituting a dynamic image, pixel values ​​between frames that are consecutive in time are compared, and based on the comparison results, it is determined whether there is a scene change. In addition, in recent years, methods using learning models that have been pre-machined to detect scene changes have also been developed.

[0003] Patent Document 1 proposes a method of detecting scene changes in a video, counting the number of consecutive scene changes, thereby detecting fast-moving scenes and selecting key frames from the scenes. Prior art literature Patent Literature

[0004] Patent Document 1: Japanese Patent Application No. 2003-519946 Summary of the invention Technical problem to be solved by the invention

[0005] In the method of comparing pixel values ​​between frames to detect scenes, for example, a portion of a camera shooting a dynamic image that is shaken during shooting may be misjudged as a scene change. In addition, in the method using a learning model that has been subjected to machine learning, the accuracy of scene change detection may decrease for dynamic images shot in a condition different from the dynamic images used in the machine learning, and the cost required for introduction may increase.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing method, a computer program, and an image processing device that can be expected to detect scene changes from dynamic images with high accuracy. Technical solutions adopted to solve technical problems

[0007] In the image processing method of the first mode, the image processing device performs the following actions: detects candidates for scene change positions from a dynamic image, detects candidate frames that can serve as main frames of the scene from frames constituting the dynamic image, determines the main frames from the candidate frames, and determines the scene change position from the candidates for scene change positions based on the candidates for scene change positions and the time sequence of the main frames.

[0008] The second image processing method is based on the first image processing method. When there is no main frame between two scene change position candidates arranged in time sequence, the scene change position is determined by removing any one of the two scene change positions from the candidates.

[0009] The third image processing method is based on the first or second image processing method, and when there are candidates for scene change positions between two main frames that are arranged in time sequence and similar, the scene change position is determined by removing the scene change position from the candidates.

[0010] The image processing method of the fourth aspect is based on the image processing method of any one of the first to third aspects, calculates a statistic for each frame, and detects a candidate for a scene change position based on a difference in the statistic between two frames.

[0011] The image processing method of the fifth aspect is based on the image processing method of any one of the first to fourth aspects, calculates a hash value for each frame, and detects scene change position candidates based on a difference in hash values ​​between two frames.

[0012] The image processing method of the sixth aspect is based on the image processing method of any one of the first to fifth aspects, extracts edges from each frame, and detects candidate frames based on a change in the edge between two frames.

[0013] The image processing method of the seventh mode is based on the image processing method of any one of the first to sixth modes, extracts feature points from candidate frames, and removes candidate frames based on comparison results of feature points between multiple candidate frames, thereby determining the main frame from the candidate frames.

[0014] The image processing method of the eighth mode is based on the image processing method of any one of the first to seventh modes, establishes a correspondence between the information related to the determined scene change position and the main frame and the dynamic image and stores them, accepts the selection of the scene change position or the main frame, and plays the dynamic image based on the selected scene change position or the main frame.

[0015] The image processing method of the ninth aspect is based on the image processing method of the eighth aspect, and establishes correspondence between information related to the determined scene change position and main frame and text information related to the dynamic image and stores them.

[0016] The image processing method of the tenth mode is based on the image processing method of the ninth mode, and obtains dynamic images obtained by shooting the construction or repair work of air-conditioning related equipment, determines the scene change position and the main frame for the obtained dynamic image, and establishes correspondence between the information related to the determined scene change position and the main frame and the text information related to the air-conditioning related equipment and stores them.

[0017] The image processing method of the eleventh mode is based on the image processing method of any one of the first to tenth modes, extracts a partial dynamic image from the dynamic image based on the determined scene change position and the main frame, and generates a summary dynamic image by combining the extracted partial dynamic image.

[0018] The image processing method of the twelfth mode is based on the image processing method of any one of the first mode to the eleventh mode, inputs the determined main frame into a learning model that classifies the type of construction or repair based on the input of the main frame of a dynamic image of the construction or repair work of air-conditioning related equipment, obtains the classification result output by the learning model, and determines the title of the dynamic image or the scene included in the dynamic image based on the obtained classification result.

[0019] The computer program of the thirteenth method causes the computer to perform the following processing: detecting candidates for scene change positions from a dynamic image, detecting candidate frames that can serve as main frames of the scene from frames constituting the dynamic image, determining the main frames from the candidate frames, and determining the scene change position from the candidates for the scene change position based on the candidates for the scene change position and the temporal order of the main frames.

[0020] The image processing device of the fourteenth mode includes: a scene change position candidate detection unit, which detects candidates for scene change positions from a dynamic image; a candidate frame detection unit, which detects candidate frames that can serve as main frames of the scene from frames constituting the dynamic image; a main frame determination unit, which determines the main frame from the candidate frames; and a scene change position determination unit, which determines the scene change position from the candidates for the scene change position based on the candidates for the scene change position and the temporal order of the main frames. Effects of the Invention

[0021] In one embodiment, it is expected that scene changes can be detected from moving images with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram for explaining the outline of the information processing system according to the present embodiment. Figure 2This is a block diagram showing the configuration of a server device according to this embodiment. Figure 3 This is a block diagram showing the structure of a terminal device according to this embodiment. Figure 4 This is a schematic diagram for explaining the detection process of the scene change position and the key frame performed by the information processing system of this embodiment. Figure 5 It is a schematic diagram showing an example of an HSL histogram. Figure 6 This is a schematic diagram showing an example of the calculation result of the difference degree between frames of a moving image. Figure 7 It is a schematic diagram showing an example of edge extraction. Figure 8 This is a schematic diagram showing an example of the calculation result of the edge change rate between frames of a dynamic image. Fig. 9 FIG. 1 is a schematic diagram showing an example of key points extracted from a candidate frame. Fig.10 : is a schematic diagram showing an example of a key point matching result. Fig.11 This is a schematic diagram for explaining a method for determining a scene change position of a server device. Fig.12 This is a flowchart showing the procedure of processing executed by the server device according to the present embodiment. Fig.13 This is a flowchart showing the procedure of processing executed by the server device according to the present embodiment. Fig.14 This is a schematic diagram showing an example of a playback screen of a terminal device. Fig.15 This is a schematic diagram for explaining a learning model used by the server device of this embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, a specific example of the information processing system according to the embodiment of the present invention will be described with reference to the drawings. In addition, the present invention is not limited to these examples, but is intended to include all changes within the meaning and scope of the claims and equivalents to the claims.

[0024] <System Structure> Figure 1It is a schematic diagram for explaining the overview of the information processing system of the present embodiment. In the information processing system of the present embodiment, the operator 102 who performs operations such as the implementation or repair of the air conditioning equipment 101 uses a camera 103 included in a headset worn on his or her head to photograph the status of the operation. In addition, in the present embodiment, the camera 103 installed on a wearable device such as a headset worn by the operator 102 is used to take pictures, but it is not limited to this. The camera 103 can also be set around the air conditioning equipment 101 and the operator 102 to photograph the operation. In addition, Figure 1 The air conditioning device 101 shown is an indoor unit of an air conditioner (air conditioner), but is not limited thereto. The air conditioning device 101 may be, for example, an outdoor unit of an air conditioner, a ventilator, a circulator, an air purifier, a heating device, a dehumidifying dryer, or various other air conditioning-related devices. In addition, the camera 103 may be used to photograph operations such as construction or repair of various devices other than air conditioning-related devices, and various operations other than construction or repair of these devices may also be photographed.

[0025] The dynamic image captured by the camera 103 is provided to the server device 1. The server device 1 obtains the dynamic image captured by one or more operators, and accumulates the obtained dynamic image in a database. For example, in the case where the camera 103 has a communication function, the method of providing the dynamic image from the camera 103 to the server device 1 can adopt a method of directly sending the dynamic image from the camera 103 to the server device 1 through wired or wireless communication. In addition, in the case where the camera 103 does not have a communication function, for example, the camera 103 can store the dynamic image in a storage medium such as a memory card or an optical disk, and provide the dynamic image from the camera 103 to the server device 1 through the storage medium. In addition, for example, a terminal device such as a PC (personal computer) or a smart phone can be interposed between the camera 103 and the server device 1, and the terminal device obtains the dynamic image from the camera 103 and sends it to the server device 1. The method of providing the dynamic image from the camera 103 to the server device 1 can adopt any method.

[0026] The server device 1 can communicate with one or more terminal devices 3 via a network such as a LAN (local area network) or the Internet. The terminal device 3 is a general-purpose information processing device such as a PC or a smart phone, and in this embodiment, it is used by an unskilled user who is learning operations such as construction or repair of the air conditioning equipment 101 to view and listen to dynamic images of operations performed by skilled operators. Based on a request from the terminal device 3, the server device 1 obtains a desired dynamic image from a plurality of dynamic images stored in a database and sends it to the terminal device 3. The terminal device 3 displays (plays) the dynamic image received from the server device 1.

[0027] The server device 1 of this embodiment detects scenes (scenes or shots, etc.) and key frames (main frames) from the dynamic images acquired by the camera 103, and stores the information related to their detection results in the database together with the dynamic images. The dynamic images captured by the camera 103 are composed of about dozens of frames (still images) in a second, and the key frames are frames that store important information related to the scenes among these multiple frames. The server device 1 sends the dynamic images and the information related to the scenes and key frames to the terminal device 3. The terminal device 3 can receive the dynamic images and the information related to the scenes and key frames from the server device 1, and, for example, accept the user's selection of scenes or key frames when playing the dynamic images, and start playing the dynamic images from the selected scenes or key frames.

[0028] The server device 1 of the present embodiment first performs a process of detecting candidates for scene change positions from the dynamic image captured by the camera 103. In addition, the server device 1 performs a process of detecting candidate frames that can serve as key frames from the dynamic image captured by the camera 103. Next, the server device 1 performs a process of determining key frames from a plurality of candidate frames detected from the dynamic image, for example, by removing similar candidate frames. Next, the server device 1 performs a process of determining the scene change position of the dynamic image from a plurality of candidates for scene change positions detected from the dynamic image, for example, based on whether the key frame exists between two consecutive candidates for scene change positions. By detecting scene change positions and key frames from the dynamic image through the above steps, it can be expected that the server device 1 can detect scene change positions and key frames suitable for the dynamic image with high accuracy.

[0029] <Device Structure> Figure 2 1 is a block diagram showing the structure of the server device 1 of the present embodiment. The server device 1 of the present embodiment is configured to include a processing unit 11, a storage unit (memory) 12, and a communication unit (transceiver) 13. In addition, in the present embodiment, the description is based on the premise that the processing is performed by one server device, but the processing may also be performed in a distributed manner by a plurality of server devices.

[0030] The processing unit 11 is composed of a CPU (central processing unit), an MPU (microprocessor), a GPU (graphics processing unit), a quantum processor, or other computing devices, a ROM (read-only memory), a RAM (random access memory), etc. The processing unit 11 reads and executes a server program 12a stored in the storage unit 12, thereby performing various processes such as detecting scene change positions and key frames from the dynamic image captured by the camera 103 and providing the dynamic image stored in the database to the terminal device 3.

[0031] The storage unit 12 is configured using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In the present embodiment, the storage unit 12 stores a server program 12a executed by the processing unit 11. In addition, a dynamic image DB (database) 12b that accumulates dynamic images captured by the camera 103 is provided in the storage unit 12.

[0032] In the present embodiment, the server program (program product) 12a is provided in the form of being stored in a storage medium 99 such as a memory card or an optical disk, and the server device 1 reads the server program 12a from the storage medium 99 and stores it in the storage unit 12. However, the server program 12a may be written to the storage unit 12, for example, during the manufacturing stage of the server device 1. In addition, for example, the server program 12a may be distributed by other remote server devices and obtained by the server device 1 through communication. For example, the server program 12a may be read by a writing device from the server program 12a stored in the storage medium 99 and written to the storage unit 12 of the server device 1. The server program 12a may be provided in the form of being distributed via a network, or may be provided in the form of being stored in the storage medium 99.

[0033] The moving image DB 12b is a database that stores and accumulates moving images captured by the camera 103. In addition, the moving image DB 12b stores information on scene change positions and key frames detected from the moving images, which are associated with the moving images.

[0034] The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (local area network), and the Internet. In the present embodiment, the communication unit 13 communicates between one or more terminal devices 3 and the camera 103 via the network N. The communication unit 13 sends data provided from the processing unit 11 to other devices, and provides data received from other devices to the processing unit 11.

[0035] In addition, the storage unit 12 may also be an external storage device connected to the server device 11. In addition, the server device 1 may also be a multi-computer configured to include a plurality of computers, or a virtual machine virtually constructed by software. In addition, the server device 1 is not limited to the above structure, and may include, for example, a reading unit for reading information stored in a portable storage medium, an input unit for receiving an operation input, or a display unit for displaying an image.

[0036] In the server device 1 of the present embodiment, the server program 12a stored in the storage unit 12 is read out and executed by the processing unit 11, and the scene change position candidate detection unit 11a, the candidate frame detection unit 11b, the key frame determination unit 11c, the scene change position determination unit 11d, and the DB processing unit 11e are implemented as software functional units in the processing unit 11. In addition, in this figure, as the functional units of the processing unit 11, the functional units related to dynamic images are illustrated, and the functional units related to other processing are omitted.

[0037] The scene change position candidate detection unit 11a performs processing to detect candidates for positions where scenes (scenes or shots, etc.) have changed from the dynamic image captured by the camera 103. The scene change position candidate detection unit 11a compares two temporally consecutive frames among the frames constituting the dynamic image, and calculates a value representing the difference between the two frames. When the calculated value exceeds a predetermined threshold, the scene change position candidate detection unit 11a determines that a scene change has occurred in the dynamic image, and uses the earlier frame of the two frames as the last frame of the scene, and the later frame as the first frame of the next scene. The scene change position candidate detection unit 11a uses the position of the first (or last) frame of the scene as a candidate for the scene change position of the dynamic image, and stores information representing the position (such as the time or number of frames from the beginning of the dynamic image to the position).

[0038] In addition, in the present embodiment, the scene change position candidate detection unit 11a calculates two types of values ​​as values ​​representing the difference between two frames. The first value representing the difference between two frames is the "Bhattacharyya distance". The Bhattacharyya distance is one of the metrics used to calculate the distance between two probability distributions. The scene change position candidate detection unit 11a creates a histogram of HSL (hue, saturation, brightness) of multiple pixel values ​​included in each of the two frames being compared. The scene change position candidate detection unit 11a calculates the Bhattacharyya distance representing the difference between the two histograms based on the two HSL histograms created for the two frames. The scene change position candidate detection unit 11a can use the positions of the two frames as candidates for the scene change position when the calculated Bhattacharyya distance exceeds a predetermined threshold.

[0039] The second value representing the difference between the two frames is the "pHash (Perseptual Hash) distance". A value of a specified length (e.g., 64 bits, 256 bits, etc.) calculated by performing a specified operation on the input information is a hash value, and pHash is a hash value having the characteristics of the input image. The scene change position candidate detection unit 11a calculates pHash from each of the two frames, and calculates the distance (e.g., Hamming distance) between the two calculated pHashes. The scene change position candidate detection unit 11a can use the positions of the two frames as candidates for the scene change position when the calculated distance of the pHash exceeds a predetermined threshold.

[0040] The scene change position candidate detection unit 11a can detect scene change position candidates based on the Bhattacharyya distance and scene change position candidates based on the pHash distance, and use the scene change position candidates detected as scene change position candidates by at least one of the methods as the final scene change position candidates. In addition, the scene change position candidate detection unit 11a can also use the scene change position candidates detected as scene change position candidates by both methods as the final scene change position candidates. In addition, the scene change position candidate detection unit 11a can also perform only one of the scene change position candidates based on the Bhattacharyya distance and the scene change position candidates based on the pHash distance, and can also detect scene change position candidates by methods other than these two methods. In addition, the scene change position candidate detection unit 11a can also calculate the average value or total value of the calculated Bhattacharyya distance and pHash distance as the difference, and when the calculated difference exceeds a threshold, use the positions of the two frames as scene change position candidates.

[0041] The candidate frame detection unit 11b detects candidate frames that can be key frames from the dynamic image captured by the camera 103. The candidate frame detection unit 11b extracts edges from each frame constituting the dynamic image, compares the edges of two frames that are consecutive in time sequence, and calculates the rate of change of the edges of the two frames. When the calculated rate of change of the edge is less than a predetermined threshold value (i.e., the change of the edge is small), the candidate frame detection unit 11b selects the frame that is earlier in time sequence (or later in time sequence) of the two frames as a candidate frame that can be a key frame.

[0042] The key frame determination unit 11c performs a process of determining the key frames of the dynamic image from the candidate frames detected by the candidate frame detection unit 11b. The key frame determination unit 11c extracts feature quantities for the multiple candidate frames detected from the dynamic image, and searches for similar candidate frames from the multiple candidate frames by comparing the features. The key frame determination unit 11c retains any candidate frame from the multiple similar candidate frames as the final key frame, and removes the other candidate frames. For example, when there are two similar candidate frames, the key frame determination unit 11c can retain the candidate frame that is earlier in time sequence and remove the candidate frame that is later in time sequence. For example, when there are three similar candidate frames, the key frame determination unit 11c can retain the candidate frame that is in the middle in time sequence and remove the candidate frames before and after.

[0043] Furthermore, in the present embodiment, in order to extract feature quantities from each candidate frame, the key frame determination unit 11c extracts key points using ORB (full name in English: Oriented FAST and Rotated BRIEF). The key frame determination unit 11c can, for example, match the key points extracted from the two candidate frames, calculate the number or ratio of key points that are consistent between the two candidate frames, and determine whether the two candidate frames are similar based on whether the value exceeds a threshold. In addition, the key frame determination unit 11c can also extract feature quantities other than the key frames of ORB and determine whether the candidate frames are similar.

[0044] The scene change position determination unit 11d performs the following processing: based on the key frame determined by the key frame determination unit 11c, the final scene change position is determined from the scene change position candidates detected by the scene change position candidate detection unit 11a. The scene change position determination unit 11d investigates the temporal order relationship between the scene change position candidates and the key frame, and determines whether the key frame exists between two scene change position candidates that are one before and one after each other in the temporal order. In this embodiment, it is a condition that at least one key frame is included in the scene determined by the scene change position (i.e., the scene presented in the dynamic image from the previous scene change position to the next scene change position). When the key frame does not exist between the two scene change position candidates that are one before and one after each other, the scene change position determination unit 11d determines that at least one of the scene change position candidates is inappropriate and removes it. The scene change position determination unit 11d repeatedly performs the above processing on all the scene change position candidates detected from the dynamic image to remove the inappropriate candidates, and determines the scene change position candidate that is finally left as the final scene change position.

[0045] Alternatively, the scene change position determination unit 11d may determine the candidate scene change position detected by the scene change position candidate detection unit 11a as the final scene change position. In this case, when no key frame exists between the two scene change positions, the scene change position determination unit 11d removes the dynamic image (i.e., the scene) between the two scene change positions from the entire dynamic image. In addition, the removal of the scene may be performed, for example, by removing the data of the scene from the data of the dynamic image and generating the data of the dynamic image with a shortened playback time. In addition, for example, the dynamic image data itself may not be changed, but the information related to the corresponding scene may be removed from the composition information of the scene stored in association with the dynamic image.

[0046] The DB processing unit 11e associates the dynamic image captured by the camera 103 with the scene change position determined by the scene change position determination unit 11d and the information related to the key frame determined by the key frame determination unit 11c for the dynamic image, and stores them in the dynamic image DB 12b of the storage unit 12. In addition, the DB processing unit 11e receives a request for playing a dynamic image from the terminal device 3, reads the data of the dynamic image requested to be played from the dynamic image DB 12b, and sends the read dynamic image and the information related to the scene change position and the key frame associated with the dynamic image to the terminal device 3 of the request source.

[0047] Figure 3 3 is a block diagram showing the structure of the terminal device 3 of the present embodiment. The terminal device 3 of the present embodiment includes a processing unit 31, a storage unit (memory) 32, a communication unit (transceiver) 33, a display unit (display) 34, an operation unit 35, etc. The terminal device 3 is a device used by a user who is an unskilled person learning the technology of construction or repair of the air conditioner 101, and can be configured using an information processing device such as a smartphone, a tablet terminal device, or a personal computer.

[0048] The processing unit 31 is composed of a CPU or MPU, a ROM, a RAM, etc. The processing unit 31 reads and executes a program 32a stored in the storage unit 32 to perform processing such as searching for moving images stored in the moving image DB 12b of the server device 1 and displaying (playing) these moving images.

[0049] The storage unit 32 is constituted by, for example, a non-volatile storage element such as a flash memory or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for the processing of the processing unit 31. In the present embodiment, the storage unit 32 stores the program 32a executed by the processing unit 31. In the present embodiment, the program 32a is distributed by a remote server device, etc., and the terminal device 3 obtains the program 32a through communication and stores it in the storage unit 32. However, the program 32a can also be written into the storage unit 32, for example, at the manufacturing stage of the terminal device 3. For example, the program 32a can also be read out by the terminal device 3 from the program 32a stored in the storage medium 98 such as a memory card or an optical disk and stored in the storage unit 32. For example, the program 32a can also be read out by a writing device from the program 32a stored in the storage medium 98 and written into the storage unit 32 of the terminal device 3. The program 32a can be provided in the form of distribution via a network, or in the form of storage in the storage medium 98.

[0050] The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, and the Internet. In the present embodiment, the communication unit 33 communicates with the server device 1 via the network N. The communication unit 33 sends data provided from the processing unit 31 to other devices, and provides data received from other devices to the processing unit 31.

[0051] The display unit 34 is configured using a liquid crystal display or the like, and displays various images and texts based on the processing of the processing unit 31. The operation unit 35 receives the user's operation and notifies the processing unit 31 of the received operation. For example, the operation unit 35 receives the user's operation through an input device such as a mechanical key or a touch panel provided on the surface of the display unit 34. In addition, for example, the operation unit 35 may also be an input device such as a mouse and a keyboard, and these input devices may also be configured to be removable relative to the terminal device 3.

[0052] In addition, the terminal device 3 of the present embodiment reads out and executes the program 32a stored in the storage unit 32 through the processing unit 31, so that the retrieval processing unit 31a and the display processing unit 31b are implemented as software functional units in the processing unit 31. In addition, the program 32a can be a dedicated program in the information processing system of the present embodiment, or a general-purpose program such as an Internet browser or a web browser.

[0053] The search processing unit 31a performs search processing on a large number of dynamic images stored in the dynamic image DB 12b of the server device 1. The search processing unit 31a accepts various search conditions input by the user, and sends the received search conditions to the server device 1. The server device 1 that receives the search conditions from the terminal device 3 extracts dynamic images that meet the search conditions from the dynamic image DB 12b, and sends the summary information of the extracted dynamic images, etc. to the terminal device 3 as the search results. The terminal device 3 displays the search results received from the server device 1, and the search processing unit 31a accepts the user's selection of the dynamic image to be played based on the search results, and requests the server device 1 to send the selected dynamic image.

[0054] The display processing unit 31b performs display processing such as displaying a screen for receiving input of search conditions, displaying information sent from the server device 1 as a search result, and playing (displaying) a dynamic image. In response to a request to send a dynamic image from the search processing unit 31a, the server device 1 reads the requested dynamic image from the dynamic image DB 12b, and sends the dynamic image and various information related to the dynamic image (for example, information on scene change positions and key frames) to the terminal device 3 of the request source. The display processing unit 31b of the terminal device 3 that has received the dynamic image from the server device 1 plays the received dynamic image and displays it on the display unit 34. In addition, the display processing unit 31b receives the information sent together with the dynamic image, and based on the scene change position and key frame information included in the received information, it can perform processing such as jumping the playback position of the dynamic image to the scene change position or key frame specified by the user.

[0055] <Scene Change Position and Key Frame Detection Processing> Figure 4 1 is a schematic diagram for explaining the detection process of scene change positions and key frames performed by the information processing system of this embodiment. Figure 4 As shown in the upper part of the figure, the dynamic image processed by the information processing system of this embodiment is formed by, for example, about dozens of frames (still images) in a second. In addition, the dynamic image can be divided into multiple scenes. In this embodiment, the scene can be called a scene or a shot in the video production, which is a unit for dividing the action of a person or object in the dynamic image. In addition, in this embodiment, Figure 4 As shown in the lower part of , a scene is obtained continuously from a plurality of frames including at least one key frame, and the first frame and the last frame are processed as scene change positions.

[0056] The server device 1 of the information processing system of this embodiment obtains the dynamic image obtained by shooting the construction or repair of the air conditioning equipment 101 by the camera 103 through the communication or storage medium, and stores the obtained dynamic image in the dynamic image DB 12b. At this time, the server device 1 performs scene change position and key frame detection processing on the obtained dynamic image, and establishes a correspondence between the information related to the detected scene change position and key frame and the dynamic image and stores it in the dynamic image DB 12b.

[0057] (1) Detection of Candidates for Scene Change Positions For the dynamic image acquired from the camera 103, the server device 1 first performs a process of detecting the candidate scene change position included in the dynamic image. The server device 1 of this embodiment calculates the value representing the difference between two frames in a temporal sequence among the multiple frames constituting the dynamic image, and determines whether the calculated value exceeds a predetermined threshold. When the value representing the difference between the frames exceeds the threshold, the server device 1 uses the positions of the two consecutive frames as candidates for the scene change position. In this embodiment, the server device 1 calculates two values, the Bhattacharyya distance of the HSL histogram and the distance of the hash value, as the value representing the difference between the frames.

[0058] Figure 5 : is a schematic diagram showing an example of an HSL histogram, and is a graph showing the distribution of the number of pixels in one frame as a sample with respect to the values ​​of H (hue), S (saturation), and L (lightness). Figure 5 The horizontal axis of the graph is the HSL value, and the vertical axis is the number of pixels. Figure 5 In the graph, the distribution of H is represented by a solid line, the distribution of S is represented by a dotted line, and the distribution of L is represented by a dashed line. The server device 1 calculates the HSL histogram of each frame constituting the dynamic image. The server device 1 can, for example, convert the pixel value of the frame provided as the RGB value into the HSL value, and calculate the HSL histogram by counting the number of pixels contained in the frame for each HSL value.

[0059] The server device 1 calculates the HSL histogram for all frames included in the dynamic image, and calculates, for each frame, a value representing the difference in the HSL histogram from the previous frame that is continuous in time. In the present embodiment, the server device 1 calculates the Bhattacharyya distance as a value representing the difference in the HSL histogram. In addition, the method for calculating the Bhattacharyya distance is an existing technology, so a detailed description is omitted. In addition, in the present embodiment, the server device 1 calculates the HSL histogram for each frame and calculates the Bhattacharyya distance between frames, but is not limited to this. The server device 1 may also calculate a statistical value other than the HSL histogram, and may also calculate a value other than the Bhattacharyya distance as a value representing the difference between frames.

[0060] In addition, the server device 1 calculates hash values ​​for all frames included in the dynamic image, such as pHash. pHash is obtained by performing frequency transformation on the image through discrete cosine transform, etc., extracting low-frequency components and calculating hash values. In addition, the server device 1 calculates the Hamming distance of pHash between each frame and the previous frame that is continuous in time sequence as a value representing the difference between the frames. pHash has the characteristic that the more similar two images are, the smaller the Hamming distance of pHash between the two images. In addition, the calculation method of pHash and Hamming distance is an existing technology, so detailed description is omitted. In addition, in the present embodiment, the server device 1 calculates pHash for each frame and calculates the Hamming distance of pHash between frames, but is not limited to this. The server device 1 can also calculate hash values ​​other than pHash (such as aHash (average hash) etc.), and can also calculate values ​​other than Hamming distance as values ​​representing the difference between frames.

[0061] In this way, the server device 1 of the present embodiment calculates two values, the Bhattacharyya distance of the HSL histogram and the Hamming distance of pHash, between each frame and the previous frame that is continuous in time sequence, as values ​​representing the difference between the two frames. In the present embodiment, the average value (weighted average value) of the two values ​​of the Bhattacharyya distance and the Hamming distance calculated by the server device 1 is calculated, and the calculated average value is used as the difference between the frames.

[0062] Figure 6 1 is a schematic diagram showing an example of calculation results of the difference between frames of a moving image, and is a graph showing changes in the difference during the playback time of the moving image. Figure 6 The horizontal axis of the graph is time (playing time of the dynamic image), and the vertical axis is the difference between frames. The server device 1 calculates the difference between all frames included in the dynamic image and the previous frame, and determines whether the difference exceeds a predetermined threshold. Figure 6 In FIG. 1 , the waveform shown by the solid line is the change in the difference degree, and the dotted horizontal line shows the threshold value.

[0063] The server device 1 uses the time when the calculated difference exceeds a predetermined threshold as a candidate for the scene change position. Figure 6 In FIG. 1 , the vertical line of the single-point dashed line is the time when the server device 1 considers the video change position as a candidate. Figure 6 There are three moments that are candidates for scene change positions, and there are moments other than these moments where the difference exceeds the threshold. This is because, in this embodiment, after the server device 1 detects the moment of the candidate for the scene change position, it is restricted in such a way that the candidate for the scene change position is not detected within a specified time. By setting such a restriction, it is expected that a large number of similar candidates for scene change positions can be suppressed from being detected.

[0064] (2) Detection of candidate frames The server device 1 performs a process of detecting candidate frames that are candidates for key frames from all the frames included in the moving image captured by the camera 103. In addition, the detection process of the candidate frames can be performed before, after, or simultaneously with the detection process of the above-described scene change position. The server device 1 of the present embodiment performs image processing for extracting edges on a plurality of frames constituting the moving image.

[0065] Figure 7 is a schematic diagram showing an example of edge extraction. In Figure 7 an example of an image (frame image) corresponding to one frame included in the moving image is shown on the left side. On the Figure 7 right side, regarding the edges extracted from the image of this frame, a binary image (edge image) is shown in which pixels corresponding to the edges are set to white and pixels other than the edges are set to black. The server device 1 can extract edges from the image of each frame by performing, for example, edge detection processing (Canny Edge Detection) implemented by the Canny method on each frame included in the moving image. The image processing for extracting edges from an image is an existing technique, so a detailed description thereof is omitted. In addition, the extraction of edges from the frame image by the server device 1 can also be performed by arbitrary image processing.

[0066] The server device 1 that extracts the edges of each frame included in the moving image compares the edges of two temporally consecutive frames and calculates the edge change rate. The server device 1 can compare two edge images extracted from two frame images, for example, calculate the total number of pixels where edge pixels become non-edge pixels and non-edge pixels become edge pixels, and calculate the ratio of the calculated total number to the total number of pixels in one frame, and use the calculated ratio as the edge change rate. In addition, as long as the edge change rate is an index indicating how much the edge has changed between frames, it can be any value. The above method for calculating the edge change rate is an example and is not limited thereto. The server device 1 can also calculate the edge change rate by any method.

[0067] Figure 8 is a schematic diagram showing an example of the calculation result of the edge change rate between frames of a moving image and is a graph showing the change in the edge change rate during the playback time of the moving image. Figure 8 The horizontal axis of the graph in Figure 8 In FIG. 1 , the horizontal straight line shows the threshold value.

[0068] The server device 1 uses the frame at the time when the calculated edge change rate is lower than the predetermined threshold (the time when it is lower than the threshold) as a candidate frame to be a key frame. Figure 8 In FIG. 1 , the vertical straight line indicates the time of the frame selected as a candidate frame by the server device 1. Figure 8 In FIG. 1 , three candidate frames are detected in the first half of the video, and one candidate frame is detected in the second half of the video. Figure 8 In addition to these four candidate frames, there are also moments when the edge change rate is lower than the threshold, and the server device 1 can also detect candidate frames at these moments.

[0069] (3) Determination of key frames The server device 1 performs a process of determining a final key frame by removing similar candidate frames from the candidate frames detected from all frames included in the dynamic image, for example, when there are multiple candidate frames within a specified time. In addition, the server device 1 performs the process of determining the key frame after the process of detecting the candidate frame, but may also perform the process of determining the key frame before the process of detecting the candidate of the scene change position, or may perform it after, or may perform it in parallel.

[0070] The server device 1 of the present embodiment, for example, performs a process of extracting the feature quantity of each candidate frame, compares the feature quantities of two candidate frames and calculates the similarity, and when the calculated similarity exceeds a threshold value, determines that the two candidate frames are similar. In the present embodiment, the server device 1 extracts the key points of ORB as the feature quantity of each candidate frame. ORB is a method that combines the key point detection of FAST (Fastened Segmentation Test Features) and the feature quantity descriptor of BRIEF (Binary Robust Independent Elementary Features). The above-mentioned ORB, FAST and BRIEF and other technologies are existing technologies, so detailed descriptions are omitted. In addition, the extraction of key points by the server device 1 can also be performed by methods other than ORB, such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Robust Features). In addition, the server device 1 can also extract feature quantities other than key points from the frame. Fig. 9 FIG. 1 is a schematic diagram showing an example of key points extracted from a candidate frame. Fig. 9 The two images shown are obtained by extracting key points from two similar candidate frames (candidate frame 1 and candidate frame 2), and the extracted key points are shown as circular points on the images.

[0071] Next, the server device 1 compares the key points extracted from the two candidate frames and performs a process of searching for the same key points, i.e., a so-called key point matching process. The key point matching process extracted by methods such as ORB is a conventional technology, and therefore, a detailed description is omitted in this embodiment. Fig.10 is a schematic diagram showing an example of key point matching results. Fig.10 In the example shown, by Fig. 9 The corresponding (matched) key points in the two candidate frames are connected by straight lines, showing the matching result of the key points.

[0072] The server device 1 counts the total number of key points extracted from the two candidate frames and the number of key point matches between the two candidate frames, and calculates the ratio of the number of matches to the total number of key points as the similarity. The server device 1 can determine whether the calculated similarity exceeds a predetermined threshold, and if the similarity exceeds the threshold, determine that the two candidate frames are similar.

[0073] The server device 1 determines a candidate frame to which there is no similar another candidate frame as a key frame included in the dynamic image. In addition, when the dynamic image includes multiple similar candidate frames, the server device 1 appropriately selects a candidate frame from the multiple candidate frames and determines it as a key frame, and removes one or more unselected candidate frames from the candidates for key frames. At this time, when there are two similar candidate frames, the server device 1, for example, uses the candidate frame that is earlier in time as the key frame and removes the candidate frame that is later in time from the candidates for key frames. At this time, when there are three similar candidate frames, the server device 1, for example, uses the second candidate frame in time as the key frame and removes the first and third candidate frames from the candidates for key frames. In addition, the method of selecting a candidate frame as a key frame from multiple similar candidate frames is not limited to the above method, and the server device 1 can select a key frame from multiple similar candidate frames by any method.

[0074] (4) Determination of scene change position After detecting the scene change position candidates and key frames from the dynamic image, the server device 1 performs a process of determining the scene change position from the scene change position candidates. In this embodiment, the server device 1 determines the final scene change position from the scene change position candidates based on the temporal order relationship between the scene change position candidates and the key frames, thereby determining the scene included in the dynamic image. In this embodiment, Figure 4 As shown in the lower part of , it is a condition that the scene of the dynamic image includes at least one key frame.

[0075] Fig.11This is a schematic diagram for explaining a method for determining a scene change position of the server device 1. The example shown in the figure shows that the server device 1 detects three scene change position candidates 1 to 3 and two key frames 1 and 2 by performing the above-mentioned processing on the dynamic image. In terms of time sequence, the scene change position candidate 1, the scene change position candidate 2, and the scene change position candidate 3 are arranged in the order of scene change position candidate 1, and key frame 1 and key frame 2 are arranged between scene change position candidates 2 and 3 in this order. There is no key frame between the scene change position candidates 1 and 2. A dynamic image composed of multiple frames existing between the scene change position candidates 1 and 2 is regarded as scene candidate 1, and a dynamic image composed of multiple frames (including two key frames 1 and 2) between the scene change position candidates 2 and 3 is regarded as scene candidate 2.

[0076] In this embodiment, the dynamic image includes one or more scenes, and one scene includes one or more key frames. Fig.11 The scene change position candidates 1 and 2 shown as the scene change position before and after the scene candidate 1 do not include a key frame and do not correspond to the scene of this embodiment. In the case where the key frame is not included between two scene change position candidates that are consecutive in time sequence, the server device 1 removes either of the two scene change position candidates and connects the scene candidate that does not include the key frame to the scene candidate that includes the key frame. Fig.11 In the example shown, the server device 1 removes candidate 2 for the scene change position, thereby connecting scene candidates 1 and 2 before and after candidate 2 for the scene change position as one scene, and uses candidates 1 and 3 for the scene change position as the final scene change position.

[0077] In the above example, for scene candidate 1 that does not include a key frame, server device 1 removes candidate 2 for a scene change position that is later in time and connects scene candidate 1 to scene candidate 2 that is later in time, but the present invention is not limited to this. Server device 1 may, for example, remove candidate 1 for a scene change position that is earlier in time and connect scene candidate 1 to a scene candidate that is before it. For scene candidates that do not include a key frame, server device 1 may remove candidates for scene change positions that are on the front side in time or candidates for scene change positions that are on the back side. Which scene change position candidate is to be removed by server device 1 may be determined in advance, for example, and may be selected based on the length of the previous and next scene candidates or the number of key frames, etc.

[0078] In addition, the server device 1 may also remove a candidate for a scene change position when the candidate exists between two key frames that are continuous and similar in time sequence. The server device 1 can use the similarity calculated when determining a key frame from the candidate frames to determine whether two key frames are similar. The server device 1 calculates the similarity based on the matching results of key points for two key frames that are continuous in time sequence, and determines that the two key frames are similar when the calculated similarity exceeds a predetermined threshold value (but a value smaller than the threshold value when determining whether the candidate frames are similar). The server device 1 can remove a candidate for a scene change position that exists between two similar key frames, and connect the scene candidates before and after the candidate for the scene change position as one scene.

[0079] Furthermore, when no key frame exists between two scene change positions, the server device 1 may remove the scene between the two scene change positions from the entire moving image. Fig.11 In the example shown, when the key frame does not exist between the scene change position candidates 1 and 2, the server device 1 may also infer that the dynamic image between them does not contain important information, and remove the scene candidate 1 between the scene change position candidates 1 and 2 from the dynamic image. In addition, at this time, the server device 1 may remove either the scene change position candidate 1 or 2 together with the scene candidate 1.

[0080] After determining all the scene change positions and key frames included in the dynamic image, the server device 1 associates the determined scene change positions and key frames with the dynamic image and stores them in the dynamic image DB 12b. In addition, when a request to send a dynamic image stored in the dynamic image DB 12b is provided from the terminal device 3, the server device 1 reads the requested dynamic image and the scene change positions and key frames associated therewith from the dynamic image DB 12b and sends them to the terminal device 3 of the request source. The terminal device 3 can use the scene change positions and key frames received from the server device 1 together with the dynamic image, for example, to accept the user's selection of a scene and play and display the dynamic image from the received scene.

[0081] Fig.12 and Fig.131 is a flowchart showing the steps of the processing performed by the server device 1 of the present embodiment. The processing unit 11 of the server device 1 of the present embodiment communicates with the camera 103, for example, through the communication unit 13, thereby acquiring a dynamic image of the construction or repair of the air conditioning equipment 101 captured by the camera 103 (step S1). The scene change position candidate detection unit 11a of the processing unit 11 calculates the HSL histogram of each frame included in the dynamic image acquired in step S1 (step S2). The scene change position candidate detection unit 11a calculates the Bhattacharyya distance between the HSL histograms of two frames consecutive in time series based on the HSL histogram of each frame calculated in step S2 (step S3).

[0082] Next, the scene change position candidate detection unit 11a calculates the pHash of each frame included in the dynamic image acquired in step S1 (step S4). The scene change position candidate detection unit 11a calculates the Hamming distance between the pHash of two frames consecutive in time series based on the pHash of each frame calculated in step S4 (step S5). The scene change position candidate detection unit 11a calculates the difference between two frames consecutive in time series based on the Bhattacharyya distance calculated in step S3 and the Hamming distance calculated in step S5 (step S6). The difference can be, for example, the total value or average value of the Bhattacharyya distance and the Hamming distance. The scene change position candidate detection unit 11a compares the difference calculated in step S6 with a specified threshold value, and detects two frames (or two frames) whose difference exceeds the threshold value as candidates for the scene change position (step S7).

[0083] In addition, the candidate frame detection unit 11b of the processing unit 11 extracts the edge of each frame included in the dynamic image acquired in step S1 (step S8). The candidate frame detection unit 11b calculates the change rate of the edge between each frame and the previous frame of the frame in time sequence based on the edge extracted in step S8 (step S9). The candidate frame detection unit 11b compares the change rate of the edge calculated in step S9 with a predetermined threshold value, and detects the frame whose edge change rate does not exceed the threshold value as a candidate frame to become a key frame (step S10).

[0084] The key frame determination unit 11c of the processing unit 11 extracts the key points of the ORB for each candidate frame detected in step S10 (step S11). The key frame determination unit 11c matches the key points between the candidate frames based on the key points of each candidate frame extracted in step S11 (step S12). The key frame determination unit 11c calculates the similarity between the candidate frames based on the key point matching result of step S12 (step S13). The key frame determination unit 11c selects one frame from the multiple candidate frames as a key frame for multiple candidate frames whose similarity exceeds a threshold, and removes the other candidate frames to determine the key frame (step S14).

[0085] The scene change position determination unit 11d of the processing unit 11 searches for a scene candidate that does not include a key frame from the scene candidates that exist between two scene change position candidates that are consecutive in time, based on the scene change position candidates detected in step S7 and the key frame determined in step S14 (step S15). The scene change position determination unit 11d removes the scene change position candidate of either of the two scene change position candidates before and after the scene candidate that does not include a key frame that is limited in step S15 (step S16). The scene change position determination unit 11d determines the scene change position candidate that is not removed in step S16 as the final scene change position (step S17). The DB processing unit 11e of the processing unit 11 associates the dynamic image acquired in step S1 with the scene change position determined in step S17 and the information related to the key frame determined in step S14, and stores them in the dynamic image DB 12b (step S19), and ends the processing.

[0086] <Playback Processing of Dynamic Images> In the information processing system of the present embodiment, the server device 1 stores the information related to the scene change position and key frame of the dynamic image determined by the above-mentioned processing in the dynamic image DB 12b in a manner corresponding to the dynamic image. In addition, various information such as the character string such as the title or description of the dynamic image given by the photographer of the dynamic image, the shooting date of the dynamic image, and the shooting location of the dynamic image are stored in the dynamic image DB 12b in a manner corresponding to the dynamic image. In addition, in the present embodiment, the dynamic image obtained by shooting the construction or repair operation of the air conditioner 101 is stored in the dynamic image DB 12b, and the text information such as the name or product number of the air conditioner 101 that is the object of the construction or repair operation in the dynamic image is stored in a manner corresponding to the dynamic image.

[0087] The terminal device 3 receives a character string as a keyword input by the user, for example, and transmits the received character string to the server device 1 to request a search for a moving image. In response to the request from the terminal device 3, the server device 1 retrieves moving images including the character string of the provided keyword such as the title, description, name or product number of the air conditioner 101 from the moving image DB 12b, and transmits information related to the matching moving images as a search result to the terminal device 3 of the request source. The terminal device 3 that receives the search result from the server device 1 displays a list of information such as the title and shooting date of the moving images matching the input keyword.

[0088] Furthermore, the terminal device 3 receives the user's selection of a dynamic image to be played from the dynamic images displayed as a list of search results, and requests the server device 1 to transmit the selected dynamic image. In response to the request from the terminal device 3, the server device reads the selected dynamic image and the information on the scene change position and key frame associated with the dynamic image from the dynamic image DB 12b and transmits the information to the terminal device 3 of the request source. The terminal device 3 that has received the dynamic image and the information on the scene change position and key frame from the server device 1 displays the playback screen of the dynamic image on the display unit 34, and plays and displays the dynamic image on the screen.

[0089] Fig.14 : is a schematic diagram showing an example of a playback screen of the terminal device 3. In the playback screen shown in the figure, a dynamic image display area for displaying dynamic images is set in the center of the upper side of the screen, and four operation buttons are arranged horizontally below the area. The four operation buttons are, from left to right, a button for returning to the beginning of the scene (scene) (return button), a button for playing dynamic images (play button), a button for skipping to the next scene (skip button), and a button for stopping the playback of dynamic images (stop button). The terminal device 3 accepts the user's operation of these operation buttons and performs processing such as playing and stopping the dynamic image. In addition, the terminal device 3 performs scene switching of the dynamic image realized by the return button and the skip button based on the scene change position received from the server device 1. For example, when the return button is operated, the terminal device 3 starts playing from the scene change position closest to the current playback time of the dynamic image. In addition, for example, when the skip button is operated, the terminal device 3 starts playing from the scene change position closest to the current playback time of the dynamic image.

[0090] In addition, in the playback screen, the terminal device 3 displays a progress bar indicating the playback time below the four operation buttons, and the progress bar shows the segments indicating the scene change positions of the dynamic image. The illustrated example is a case where the dynamic image includes three scenes, and two segments of thick vertical lines indicating the video change positions are shown in the progress bar. In addition, the terminal device 3 displays the reduced images of one or more key frames included in the dynamic image below the progress bar with appropriate configuration. The terminal device 3 shows the time when these key frames appear in the dynamic image by arrows connecting the images of the key frames with the progress bar. The arrow indicates the time position where the corresponding key frame appears in the playback time of the dynamic image shown in the progress bar. When the terminal device 3 receives an operation such as a click or tap on the key frame displayed in the playback screen, it starts playing the dynamic image from the time corresponding to the key frame that received the operation.

[0091] in addition, Fig.14 The method of displaying the moving images shown is an example and is not limited thereto, and the terminal device 3 can display the moving images by any method.

[0092] <Overview of the dynamic image generation process> In the information processing system of the present embodiment, the server device 1 performs processing for generating a summary dynamic image (summary dynamic image) of the dynamic image based on the scene change positions and key frames of the dynamic image determined by the above processing. The server device 1 extracts (cuts out) one or more partial dynamic images from the dynamic image and connects them, thereby generating a summary dynamic image with a shorter playback time than the original dynamic image.

[0093] The server device 1 of this embodiment extracts partial dynamic images of a predetermined time (e.g., several seconds to several tens of seconds) from the scene change position and partial dynamic images of a predetermined time before and after the key frame from the entire dynamic image. The server device 1 connects the above-mentioned plurality of partial dynamic images extracted from the dynamic image in a time-series order to generate a digest dynamic image. The server device 1 associates the generated digest dynamic image with the original dynamic image and stores them in the dynamic image DB 12b.

[0094] The server device 1 may generate a summary dynamic image when acquiring a dynamic image from the camera 103, or may generate a summary dynamic image in response to a request from the terminal device 3, or may generate a summary dynamic image at a time other than the above. In addition, when sending a search result in response to a search request for dynamic images from the terminal device 3, the server device 1 may read a summary dynamic image of a dynamic image that matches the search result from the dynamic image DB 12b and send it to the terminal device 3. The terminal device 3 may also display the summary dynamic image together with information such as titles of a plurality of dynamic images that match the search condition as a list of dynamic image search results.

[0095] <Title Generation Process> In the information processing system of this embodiment, the server device 1 can use a learning model that has undergone machine learning, namely the so-called AI (artificial intelligence) to automatically generate the title of a dynamic image, the title of a scene included in a dynamic image, or the title of a key frame included in a dynamic image, etc. Fig.15 This is a schematic diagram for explaining the learning model used by the server device 1 of this embodiment. The learning model used by the server device 1 of this embodiment is a learning model that has been preliminarily machine-learned in order to accept a key frame included in a dynamic image as an input and output the type of construction or repair of the air-conditioning equipment 101 presented in the key frame as a classification result.

[0096] The learning model of this embodiment can be a learning model with a structure such as CNN (convolutional neural network) or DNN (deep neural network). The learning model can be generated by so-called supervised machine learning using learning data corresponding to an image and a type of construction or repair presented in the image. The supervised learning process of the learning model is an existing technology, so a detailed description is omitted, but the server device 1 can learn the learning model by methods such as gradient descent method, probabilistic gradient descent method or error back propagation method.

[0097] The server device 1 stores the learning model generated by machine learning in the storage unit 12. After detecting the key frames included in the dynamic image, the server device 1 inputs the detected one or more key frames into the learning model respectively, and obtains the classification results output by the learning model. The category of construction or repair of the air-conditioning equipment 101 output by the learning model is, for example, the construction of the outdoor unit of the air conditioner (air conditioner) or the repair of the indoor unit of the air conditioner. For example, the server device 1 classifies the category of construction or repair using the learning model for each key frame included in the dynamic image, obtains the classification results of all the key frames, and generates the title of the dynamic image and the title of the scene based on the obtained classification results.

[0098] The server device 1 appropriately selects a classification result from the classification results of multiple key frames included in the dynamic image, for example, selects the classification result with the most. For example, the server device 1 can use a character string such as "XXXX year YY month ZZ day: construction of air conditioner outdoor unit" obtained by combining the selected classification result with the time when the dynamic image was taken as the title of the dynamic image. In addition, the server device 1 can appropriately select a classification result of a key frame included in each scene included in the dynamic image, and use a character string such as "repair of air conditioner indoor unit" equivalent to the selected classification result as the title of the scene. In addition, the method of generating a title based on the classification result of the construction or repair category implemented by the server device 1 is not limited to the above method, and any method can also be used.

[0099] In addition, as an alternative to a learning model that classifies the type of construction or repair of the air conditioning equipment 101 presented in the key frame, the server device 1 may also use a learning model that receives an input of a key frame and generates a character string of the title of the key frame. Such a learning model can be constructed by combining a learning model such as CNN that converts an image into a feature quantity with a learning model such as RNN (recurrent neural network), LSTM (long short-term memory), BERT (Bidirectional Encoded Representation based on Transformer), or GPT-3 (Generative Pre-trained Transformer-3) that generates a character string of the title based on the feature quantity.

[0100] The server device 1 associates the title of the moving image generated based on the key frame and the title of the scene included in the moving image with the moving image and stores them in the moving image DB 12b. In addition, for example, when the photographer of the moving image or the like inputs the title of the moving image or the scene, the server device 1 may store the input title in the moving image DB 12b without generating the title using the learning model. In addition, the terminal device 3 may, for example, Fig.14 In the playback screen shown, the title of the dynamic image is displayed at the top, and the title of the scene is displayed in a manner associated with the image of one or more key frames.

[0101] Summary In the information processing system of the present embodiment having the above structure, the server device 1 detects candidates for scene change positions from the dynamic image captured by the camera 103, detects candidate frames that can become key frames (main frames) of the scene from multiple frames constituting the dynamic image, determines the key frame from the candidate frames, and determines the scene change position from the candidates for scene change positions based on the candidates for scene change positions and the temporal order of the key frames. Thus, the information processing system of the present embodiment can be expected to detect scene changes from dynamic images with high accuracy.

[0102] Furthermore, in the information processing system of the present embodiment, when a key frame does not exist between two candidates for scene change positions arranged in time sequence, the server device 1 determines the scene change position by removing any one of the two candidates for scene change positions. Furthermore, in the information processing system of the present embodiment, when a candidate for scene change position exists between two key frames that are continuous and similar in time sequence, the server device 1 can also determine the scene change position by removing the candidate for scene change position. Thus, the information processing system of the present embodiment can expect to determine the scene change position with high accuracy from the candidates for scene change positions detected in the dynamic image.

[0103] In addition, in the information processing system of the present embodiment, the server device 1 calculates the statistical value (HSL histogram) of each frame included in the dynamic image, and detects the candidate of the scene change position based on the difference (difference) of the statistical value of two frames consecutive in time. In addition, in the information processing system of the present embodiment, the server device 1 calculates the hash value (pHash) of each frame included in the dynamic image, and detects the candidate of the scene change position based on the difference of the hash value of two frames consecutive in time. As a result, the information processing system of the present embodiment can expect to detect the candidate of the scene change position from the dynamic image with high accuracy.

[0104] In addition, in the information processing system of the present embodiment, the server device 1 extracts edges from each frame included in the dynamic image, and detects candidate frames based on the change (rate of change) of the edge between two frames consecutive in time. In addition, in the information processing system of the present embodiment, the server device 1 extracts feature points (key points) from the candidate frames, and removes similar candidate frames based on the comparison results of the feature points between multiple candidate frames, thereby determining key frames from the candidate frames. As a result, the information processing system of the present embodiment can expect to detect candidate frames from the dynamic image with high accuracy, and determine key frames from the candidate frames with high accuracy.

[0105] In addition, in the information processing system of the present embodiment, the server device 1 associates the information related to the determined scene change position and key frame with the dynamic image and stores it in the dynamic image DB 12b. The terminal device 3 accepts the user's selection of the scene change position or key frame, and plays the dynamic image based on the accepted scene change position or key frame. Thus, the information processing system of the present embodiment can expect to play the dynamic image from the scene required by the user.

[0106] In addition, in the information processing system of the present embodiment, the server device 1 associates the information related to the determined scene change position and key frame with the text information such as the title, description, shooting date, shooting location, name or product number of the air conditioner (air conditioner related equipment) 101 to be repaired or repaired of the dynamic image, and stores them in the dynamic image DB 12b. Thus, the user can search for the dynamic image by inputting a keyword based on the text information associated with the dynamic image.

[0107] Furthermore, in the information processing system of the present embodiment, partial dynamic images are extracted from the dynamic images based on the determined scene change positions and key frames, and the extracted partial dynamic images are combined to generate a summary dynamic image (abstract dynamic image). Thus, the information processing of the present embodiment can provide the user with a summary dynamic image, and the user can expect to easily grasp the summary of a dynamic image using the summary dynamic image even if the playback time is long.

[0108] Furthermore, in the information processing system of the present embodiment, the server device 1 inputs the determined key frame into a learning model that has been previously subjected to machine learning, and determines the title of the dynamic image or the scene included in the dynamic image based on the information output by the learning model. Thus, even if the photographer of the dynamic image does not input a title, the information processing system of the present embodiment can automatically assign a title to the dynamic image.

[0109] The embodiments disclosed this time should be considered as illustrative in all aspects and not restrictive. The scope of the present invention is indicated by the claims rather than the above meaning, and is intended to include all changes within the meaning and scope equivalent to the claims. Explanation of symbols

[0110] 1 server device (image processing device); 3. Terminal device; 11. Processing Department; 11a scene change position candidate detection unit; 11b candidate frame detection unit; 11c key frame determination unit (main frame determination unit); 11d scene change position determination unit; 11e DB processing department; 12 storage unit; 12a Server programs (computer programs); 12b dynamic image DB; 13 Ministry of Communications; 31 Processing Department 31a search processing unit; 31b display processing unit; 32 storage unit; 32a Procedure; 33 Ministry of Communications; 34 display unit; 35 operation unit; 101 Air conditioning equipment (air conditioning related equipment); 102 operators; 103 cameras; N network.

Claims

1. An image processing method, characterized in that: The image processing device performs the following actions: Detecting scene change position candidates from dynamic images, Detecting candidate frames that can become main frames of the scene from frames constituting the dynamic image, determining a primary frame from the candidate frames, Based on the candidates for the scene change position and the temporal order of the key frames, the scene change position is determined from among the candidates for the scene change position.

2. The image processing method according to claim 1, characterized in that: When there is no major frame between two scene change position candidates arranged in time series, the scene change position is determined by removing either of the two scene change position candidates.

3. The image processing method according to claim 1 or 2, characterized in that: When there are candidates for a scene change position between two main frames that are aligned and similar in time series, the scene change position is determined by removing the scene change position from the candidates.

4. The image processing method according to any one of claims 1 to 3, characterized in that: Calculate the statistics of each frame, Candidates for scene change positions are detected based on the difference in statistical values ​​between two frames.

5. The image processing method according to any one of claims 1 to 4, characterized in that: Calculate the hash value of each frame, Candidates for scene change positions are detected based on the difference in hash values ​​between two frames.

6. The image processing method according to any one of claims 1 to 5, characterized in that: Extract edges from each frame, Candidate frames are detected based on the changes in the edges between two frames.

7. The image processing method according to any one of claims 1 to 6, characterized in that: Extract feature points from candidate frames, The candidate frames are removed based on the comparison results of the feature points among the plurality of candidate frames, thereby determining the main frame from the candidate frames.

8. The image processing method according to any one of claims 1 to 7, characterized in that: The information related to the determined scene change position and the main frame is associated with the dynamic image and stored. Accepts the selection of a scene change position or key frame, The dynamic image is played based on the selected scene change position or main frame.

9. The image processing method according to claim 8, characterized in that: The information related to the determined scene change position and the main frame and the text information related to the dynamic image are associated and stored.

10. The image processing method according to claim 9, characterized in that: Acquire dynamic images obtained by shooting the construction or repair work of air-conditioning related equipment, Determine the scene change position and main frame for the acquired dynamic image. The information related to the determined scene change position and the main frame and the text information related to the air-conditioning related equipment are associated and stored.

11. The image processing method according to any one of claims 1 to 10, characterized in that: extracting a portion of the dynamic image from the dynamic image based on the determined scene change position and the main frame, The extracted partial dynamic images are combined to generate a summary dynamic image.

12. The image processing method according to any one of claims 1 to 11, characterized in that: The determined main frame is input to a learning model, and the learning model classifies the type of construction or repair for the input of the main frame of the dynamic image in which the construction or repair work of the air-conditioning related equipment is captured, Get the classification results output by the learning model. The title of the dynamic image or the scene included in the dynamic image is determined based on the acquired classification result.

13. A computer program, characterized in that The computer program causes the computer to execute the following processing: Detecting scene change position candidates from dynamic images, Detecting candidate frames that can become main frames of the scene from frames constituting the dynamic image, determining a primary frame from the candidate frames, Based on the candidates for the scene change position and the temporal order of the key frames, the scene change position is determined from among the candidates for the scene change position.

14. An image processing device, characterized in that: include: a scene change position candidate detection unit for detecting scene change position candidates from a dynamic image; a candidate frame detection unit for detecting a candidate frame that can become a main frame of a scene from frames constituting the dynamic image; a main frame determination unit that determines a main frame from among the candidate frames; as well as A scene change position determination unit determines a scene change position from among the scene change position candidates based on the scene change position candidates and the temporal order of the main frames.

Images