Image processing method, computer program product, and image processing apparatus

Abstract

Provided is an image processing method, computer program product, and image processing apparatus capable of accurately detecting a scene change from a moving image. In the image processing method of the present embodiment, the image processing apparatus detects candidates for a scene change position from a moving image, detects candidate frames that are main frames of a scene from frames that constitute the moving image, determines a main frame from the candidate frames, and determines a scene change position from the candidates for a scene change position based on the order in time series of the main frame and the candidates for a scene change position. In a case where there is no main frame between two candidates for a scene change position arranged in time series, the scene change position can be determined by removing either of the two scene change positions from the candidates.

Description

Technical Field

[0001] This invention relates to an image processing method, computer program product, and image processing apparatus for performing image processing on moving images. Background Technology

[0002] Image processing techniques for detecting changes in scenes (scenes or shots, etc.) in moving images are widely used. In such techniques, for example, pixel values ​​between temporally consecutive frames constituting a moving image are compared, and the results of this comparison are used to determine if there has been a change in the scene. Furthermore, in recent years, methods have been developed that utilize pre-learned machine learning models designed to detect scene changes.

[0003] Patent document 1 proposes a method for detecting fast-moving scenes and selecting keyframes from a scene by detecting scene changes within a video, counting the number of consecutive scene changes, and then selecting keyframes from the scene.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Publication No. 2003-519946 Summary of the Invention

[0007] The technical problem that the invention aims to solve

[0008] In methods that compare pixel values ​​between frames for scene detection, there is a possibility of misjudging, for example, a camera shaking during the capture of a moving image, as a change in scene. Furthermore, in methods using machine learning models, the accuracy of scene change detection may decrease for moving images captured under conditions different from those used in the machine learning model, and the cost of implementing such models may increase.

[0009] The present invention was made in view of the above circumstances, and its object is to provide an image processing method, computer program product, and image processing apparatus that can detect changes in a scene with high precision from a moving image.

[0010] Technical solutions adopted to solve technical problems

[0011] In the first aspect of the image processing method, the image processing apparatus performs the following operations: detecting candidates for scene change locations from a moving image, detecting candidate frames that can serve as the main frames of a scene from the frames constituting the moving image, determining the main frame from the candidate frames, and determining the scene change location from the candidates for scene change locations based on the candidates for scene change locations and the temporal order of the main frames.

[0012] The second image processing method, based on the first image processing method, determines the scene change position by removing either of the two scene change positions from the candidates when there is no main frame between the two candidate scene change positions arranged in time sequence.

[0013] The third image processing method, based on the first or second image processing method, determines the scene change location by removing the scene change location from the candidates when there are candidates for scene change locations between two main frames that are temporally arranged and similar.

[0014] The fourth image processing method is based on any of the first to third methods, calculates the statistical values ​​of each frame, and detects candidates for scene change locations based on the difference between the statistical values ​​of two frames.

[0015] The fifth image processing method is based on any of the first to fourth methods, calculates the hash value of each frame, and detects candidates for scene change locations based on the difference between the hash values ​​of two frames.

[0016] The sixth image processing method is based on any of the first to fifth methods, extracting edges from each frame and detecting candidate frames based on the changes in edges between two frames.

[0017] The seventh image processing method is based on any of the image processing methods in the first to sixth methods, extracts feature points from candidate frames, and removes candidate frames based on the comparison results of feature points between multiple candidate frames, thereby determining the main frame from the candidate frames.

[0018] The eighth image processing method is based on the image processing methods of any one of the first to seventh methods. It establishes a correspondence between the information related to the determined scene change position and main frame and the dynamic image and stores it. It accepts the selection of the scene change position or main frame and plays the dynamic image based on the selected scene change position or main frame.

[0019] The ninth image processing method is based on the eighth image processing method. It establishes a correspondence between the determined scene change location and the information related to the main frame and the text information related to the dynamic image and stores them.

[0020] The tenth image processing method is based on the ninth image processing method. It acquires dynamic images obtained by capturing the construction or repair work of air conditioning-related equipment, determines the scene change positions and main frames for the acquired dynamic images, and establishes a correspondence between the determined scene change positions and main frames and the text information related to air conditioning-related equipment and stores them.

[0021] The eleventh image processing method is based on the image processing methods of any one of the first to tenth methods. Based on the determined scene change position and main frame, a portion of the dynamic image is extracted from the dynamic image, and a summary dynamic image is generated by combining the extracted portion of the dynamic image.

[0022] The twelfth image processing method is based on the image processing methods of any one of the first to eleventh methods. It inputs the determined main frame into a learning model that classifies the category of construction or repair based on the input of the main frame of the dynamic image of the construction or repair work of the air conditioning related equipment. It obtains the classification result output by the learning model and determines the title of the dynamic image or the scene included in the dynamic image based on the obtained classification result.

[0023] The computer program product of the thirteenth method causes a computer to perform the following processes: detect candidates for scene change locations from a moving image, detect candidate frames that can serve as the main frames of a scene from the frames constituting the moving image, determine the main frame from the candidate frames, and determine the scene change location from the candidates for scene change locations based on the temporal order of the candidates for scene change locations and the main frames.

[0024] The image processing apparatus of the fourteenth type includes: a scene change position candidate detection unit that detects candidates for scene change positions from a moving image; a candidate frame detection unit that detects candidate frames that can serve as main frames of a scene from frames constituting the moving image; a main frame determination unit that determines a main frame from the candidate frames; and a scene change position determination unit that determines a scene change position from the candidates based on the temporal order of the candidates for scene change positions and the main frames.

[0025] Invention Effects

[0026] In one implementation, it is expected that changes in the scene can be detected with high precision from dynamic images. Attached Figure Description

[0027] Figure 1 This is a schematic diagram used to illustrate the general outline of the information processing system of this embodiment.

[0028] Figure 2 This is a block diagram illustrating the structure of the server device according to this embodiment.

[0029] Figure 3 This is a block diagram illustrating the structure of the terminal device in this embodiment.

[0030] Figure 4 This is a schematic diagram used to explain the scene change location and keyframe detection processing performed by the information processing system of this embodiment.

[0031] Figure 5 This is a schematic diagram representing an example of an HSL histogram.

[0032] Figure 6 This is a schematic diagram illustrating an example of the calculation results for the inter-frame difference of a moving image.

[0033] Figure 7 This is a schematic diagram illustrating an example of edge extraction.

[0034] Figure 8 This is a schematic diagram illustrating an example of the calculation results of the edge change rate between frames of a dynamic image.

[0035] Figure 9 This is a schematic diagram illustrating an example of key points extracted from candidate frames.

[0036] Figure 10 This is a schematic diagram representing an example of the matching results for key points.

[0037] Figure 11 This is a schematic diagram illustrating a method for determining the location of changes in the scene of a server device.

[0038] Figure 12 This is a flowchart illustrating the steps of the processing performed by the server device in this embodiment.

[0039] Figure 13 This is a flowchart illustrating the steps of the processing performed by the server device in this embodiment.

[0040] Figure 14 This is a schematic diagram illustrating an example of the playback screen on a terminal device.

[0041] Figure 15 This is a schematic diagram used to explain the learning model used by the server device in this embodiment. Detailed Implementation

[0042] Hereinafter, specific examples of an information processing system according to embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, the present invention is not limited to these examples, but is intended to include all modifications expressed in and equivalent to the claims.

[0043] <System Structure>

[0044] Figure 1 This is a schematic diagram used to explain the overview of the information processing system of this embodiment. In the information processing system of this embodiment, the operator 102 performing operations such as servicing or repairing the air conditioning equipment 101 records the work situation using a camera 103 included in a headset or similar device worn on their head. However, while this embodiment uses a camera 103 mounted on a wearable device such as a headset worn by the operator 102 for recording, it is not limited to this; the camera 103 can also be placed around the air conditioning equipment 101 and the operator 102 to record the work. Furthermore, Figure 1 The air conditioning device 101 shown is the indoor unit of an air conditioner, but it is not limited to this. The air conditioning device 101 can also be, for example, the outdoor unit of an air conditioner, a ventilation device, a circulator, an air purifier, a heating device, or a dehumidifier, and other various air conditioning-related devices. In addition, the camera 103 can be used to film the construction or repair work of various devices other than air conditioning-related devices, and can also film various other work other than the construction or repair of these devices.

[0045] The moving images captured by camera 103 are provided to server device 1. Server device 1 acquires the moving images captured by one or more operators and accumulates the acquired moving images in a database. If camera 103 has communication capabilities, the method of providing moving images from camera 103 to server device 1 can be to directly send the moving images from camera 103 to server device 1 via wired or wireless communication. Alternatively, if camera 103 does not have communication capabilities, for example, camera 103 can store moving images on a storage medium such as a memory card or optical disc, and provide the moving images from camera 103 to server device 1 via the storage medium. Furthermore, for example, a terminal device such as a PC (personal computer) or smartphone can be positioned between camera 103 and server device 1, and the terminal device can acquire moving images from camera 103 and send them to server device 1. The method of providing moving images from camera 103 to server device 1 can be any method.

[0046] Server device 1 can communicate with one or more terminal devices 3 via a network such as a LAN (Local Area Network) or the Internet. Terminal device 3 is, for example, a general-purpose information processing device such as a PC or smartphone, which in this embodiment is used by an unskilled user learning to construct or repair air conditioning equipment 101 to capture, for example, dynamic images of skilled workers performing their work. Based on a request from terminal device 3, server device 1 retrieves the desired dynamic image from a plurality of dynamic images stored in a database and sends it to terminal device 3. Terminal device 3 displays (plays) the dynamic image received from server device 1.

[0047] In this embodiment, the server device 1 detects scenes (scenes or shots, etc.) and keyframes (main frames) from the moving images acquired by the camera 103, and stores information related to their detection results along with the moving images in a database. The moving images captured by the camera 103 consist of approximately dozens of consecutive frames (still images) within one second, and keyframes are frames in these multiple frames that store important information related to the scene. The server device 1 sends the moving images and information related to the scenes and keyframes to the terminal device 3. The terminal device 3 can receive the moving images and information related to the scenes and keyframes from the server device 1, and, for example, accept the user's selection of a scene or keyframe when playing the moving images, and start playing the moving images from the selected scene or keyframe.

[0048] The server device 1 in this embodiment first performs processing to detect candidate positions of scene changes from the moving image captured by the camera 103. Furthermore, the server device 1 performs processing to detect candidate frames that can serve as keyframes from the moving image captured by the camera 103. Next, the server device 1, for example, removes similar candidate frames, and then performs processing to determine keyframes from the multiple candidate frames detected in the moving image. Next, the server device 1, for example, determines the scene change position of the moving image from the multiple candidate positions of scene changes detected in the moving image based on whether a keyframe exists between two consecutive candidate positions of scene changes. By performing the above steps to detect scene change positions and keyframes from the moving image, it is expected that the server device 1 can detect scene change positions and keyframes suitable for the moving image with high accuracy.

[0049] <Device Structure>

[0050] Figure 2This is a block diagram illustrating the structure of the server device 1 according to this embodiment. The server device 1 of this embodiment is configured to include a processing unit 11, a storage unit (memory) 12, and a communication unit (transceiver) 13, etc. In addition, this embodiment is described under the premise that processing is performed by a single server device, but processing can also be performed by multiple server devices distributed in a decentralized manner.

[0051] The processing unit 11 is configured using a computing device such as a CPU (Central Processing Unit), MPU (Microprocessor), GPU (Graphics Processing Unit), or quantum processor, ROM (Read-Only Memory), and RAM (Random Access Memory). The processing unit 11 reads and executes the server program 12a stored in the storage unit 12, thereby performing various processes such as detecting scene changes and keyframes in the moving images captured by the camera 103, and providing the moving images stored in the database to the terminal device 3.

[0052] The storage unit 12 is configured using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for processing by the processing unit 11. In this embodiment, the storage unit 12 stores the server program 12a executed by the processing unit 11. Furthermore, a dynamic image database 12b is provided in the storage unit 12 to accumulate dynamic images captured by the camera 103.

[0053] In this embodiment, the server program (program product) 12a is provided in the form of a storage medium 99 such as a memory card or optical disc, and the server device 1 reads the server program 12a from the storage medium 99 and stores it in the storage unit 12. However, the server program 12a may also be written to the storage unit 12 during the manufacturing stage of the server device 1, for example. Furthermore, the server program 12a may be distributed by other remote server devices, for example, and obtained by the server device 1 through communication. For example, the server program 12a may also be read from the storage medium 99 by a writing device and written to the storage unit 12 of the server device 1. The server program 12a may be provided in the form of distribution via a network or in the form of storage in the storage medium 99.

[0054] The motion image database 12b is a database that stores and accumulates motion images captured by the camera 103. In addition, the motion image database 12b stores information that corresponds to these motion images and is related to the location of scene changes detected from the motion images and key frames.

[0055] The communication unit 13 communicates with various devices via a network N, including mobile phone communication networks, wireless LANs (local area networks), and the Internet. In this embodiment, the communication unit 13 communicates between one or more terminal devices 3 and the camera 103 via the network N. The communication unit 13 sends data provided by the processing unit 11 to other devices and provides data received from other devices to the processing unit 11.

[0056] Alternatively, the storage unit 12 may be an external storage device connected to the server device 11. Furthermore, the server device 1 may be configured as a multi-computer system comprising multiple computers, or it may be a virtual machine constructed virtually through software. Moreover, the server device 1 is not limited to the above-described structure; for example, it may include a reading unit for reading information stored in a portable storage medium, an input unit for receiving operational input, or a display unit for displaying images.

[0057] Furthermore, in the server device 1 of this embodiment, the server program 12a stored in the storage unit 12 is read and executed by the processing unit 11. The scene change position candidate detection unit 11a, candidate frame detection unit 11b, keyframe determination unit 11c, scene change position determination unit 11d, and DB processing unit 11e are implemented as software functional units in the processing unit 11. In this figure, functional units related to moving images are shown as functional units of the processing unit 11; functional units related to other processing are omitted from the illustration.

[0058] The scene change position candidate detection unit 11a processes candidates for locations where scenes (scenes or shots, etc.) have changed in the moving image captured by the camera 103. The scene change position candidate detection unit 11a compares two consecutive frames in time that constitute the moving image and calculates a value representing the difference between the two frames. If the calculated value exceeds a predetermined threshold, the scene change position candidate detection unit 11a determines that a scene change has occurred in the moving image, and designates the earlier frame of the two frames as the last frame of the scene, and the later frame as the first frame of the next scene. The scene change position candidate detection unit 11a uses the position of the first (or last) frame of the scene as a candidate for a scene change position in the moving image and stores information representing that position (time or number of frames from the beginning of the moving image to that position, etc.).

[0059] Furthermore, in this embodiment, the scene change position candidate detection unit 11a calculates two types of values ​​as values ​​representing the difference between two frames. The first value representing the difference between the two frames is the "Bhattacharyya Distance". The Bhattacharyya Distance is one of the measures used to calculate the distance between two probability distributions. The scene change position candidate detection unit 11a creates a histogram of HSL (hue, saturation, brightness) values ​​of multiple pixels included in the frame for each of the two frames being compared. The scene change position candidate detection unit 11a calculates the Bhattacharyya Distance representing the difference between the two histograms based on the two HSL histograms created for the two frames. The scene change position candidate detection unit 11a can consider the positions of the two frames as candidates for scene change positions if the calculated Bhattacharyya Distance exceeds a predetermined threshold.

[0060] The second value representing the difference between the two frames is the "pHash (Perseptual Hash) distance". A hash value of a predetermined length (e.g., 64 bits, 256 bits, etc.) calculated through prescribed computational processing of the input information is the hash value; the pHash is a hash value containing features of the input image. The scene change location candidate detection unit 11a calculates the pHash from each of the two frames and calculates the distance (e.g., Hamming distance) between the two calculated pHashes. The scene change location candidate detection unit 11a can consider the locations of the two frames as candidates for scene change locations if the calculated pHash distance exceeds a predetermined threshold.

[0061] The scene change position candidate detection unit 11a is capable of detecting scene change position candidates based on Bartholomew's distance and scene change position candidates based on pHash distance, and uses the scene change position candidates detected by at least one method as the final scene change position candidates. Alternatively, the scene change position candidate detection unit 11a may use the scene change position candidates detected by both methods as the final scene change position candidates. Furthermore, the scene change position candidate detection unit 11a may perform only the detection of scene change position candidates based on Bartholomew's distance and scene change position candidates based on pHash distance, or it may use methods other than these two methods to detect scene change position candidates. Additionally, the scene change position candidate detection unit 11a may calculate the average or sum of the calculated Bartholomew's distance and pHash distance as a difference score, and if the calculated difference score exceeds a threshold, the positions of the two frames are considered as scene change position candidates.

[0062] The candidate frame detection unit 11b processes the detection of candidate frames that can serve as keyframes from the dynamic image captured by the camera 103. The candidate frame detection unit 11b extracts edges for each frame constituting the dynamic image and compares the edges of two temporally consecutive frames, calculating the rate of change of the edges between the two frames. If the calculated rate of change of the edges is less than a predetermined threshold (i.e., the edge change is small), the candidate frame 11b identifies the frame that is temporally earlier (or later) of the two frames as a candidate frame that can serve as a keyframe.

[0063] The keyframe determination unit 11c performs the process of determining keyframes of the moving image from the candidate frames detected by the candidate frame detection unit 11b. The keyframe determination unit 11c extracts feature values ​​from multiple candidate frames detected from the moving image and searches for similar candidate frames from the multiple candidate frames by comparing the features. The keyframe determination unit 11c retains any one candidate frame from the multiple similar candidate frames as the final keyframe and removes the other candidate frames. For example, when two similar candidate frames exist, the keyframe determination unit 11c can retain the candidate frame that is earlier in time and remove the candidate frame that is later. For example, when three similar candidate frames exist, the keyframe determination unit 11c can retain the candidate frame that is exactly in the middle in time and remove the candidate frames before and after it.

[0064] Furthermore, in this embodiment, in order to extract feature values ​​from each candidate frame, the keyframe determination unit 11c performs keypoint extraction using ORB (Oriented Fast and Rotated BRIEF). For example, the keyframe determination unit 11c can match keypoints extracted from two candidate frames respectively, calculate the number or proportion of identical keypoints between the two candidate frames, and determine whether the two candidate frames are similar based on whether this value exceeds a threshold. Alternatively, the keyframe determination unit 11c can also extract feature values ​​other than those from the ORB keyframes and determine whether candidate frames are similar.

[0065] The scene change position determination unit 11d performs the following processing: based on the keyframe determined by the keyframe determination unit 11c, it determines the final scene change position from the candidates for scene change positions detected by the scene change position candidate detection unit 11a. The scene change position determination unit 11d investigates the temporal order relationship between the candidates for scene change positions and the keyframes, and determines whether a keyframe exists between two candidates for scene change positions that are sequentially adjacent. In this embodiment, at least one keyframe is included in the scene determined by the scene change position (i.e., the scene presented in the motion picture from the earlier scene change position to the later scene change position). If a keyframe does not exist between two candidates for scene change positions that are sequentially adjacent, the scene change position determination unit 11d determines that at least one candidate for scene change position is unsuitable and removes it. The scene change position determination unit 11d repeatedly performs the above processing on all candidates for scene change positions detected from the motion picture to remove unsuitable candidates, and determines the remaining candidates for scene change positions as the final scene change position.

[0066] Alternatively, the scene change position determination unit 11d may determine the candidate scene change positions detected by the scene change position candidate detection unit 11a as the final scene change position. In this case, if the keyframe does not exist between the two scene change positions, the scene change position determination unit 11d removes the motion picture (i.e., the scene) between the two scene change positions from the overall motion picture. Furthermore, scene removal can be performed, for example, by removing the scene data from the motion picture data and generating motion picture data with a shortened playback time. Alternatively, the motion picture data itself may remain unchanged, and information related to the corresponding scene may be removed from the scene composition information stored associated with the motion picture.

[0067] DB processing unit 11e establishes and stores in storage unit 12 a dynamic image DB 12b corresponding to the dynamic image captured by camera 103 and information related to the scene change position determined by scene change position determination unit 11d and the key frame determined by key frame determination unit 11c for the dynamic image. Furthermore, DB processing unit 11e receives a dynamic image playback request from terminal device 3, reads the data of the requested dynamic image from dynamic image DB 12b, and sends the read dynamic image and the information corresponding to the scene change position and key frame to the requesting terminal device 3.

[0068] Figure 3This is a block diagram illustrating the structure of the terminal device 3 according to this embodiment. The terminal device 3 of this embodiment is configured including a processing unit 31, a storage unit (memory) 32, a communication unit (transceiver) 33, a display unit (display) 34, and an operation unit 35. The terminal device 3 is, for example, a device used by unskilled users learning techniques such as the construction or repair of air conditioning equipment 101, and is configured using information processing devices such as smartphones, tablet computers, or personal computers.

[0069] The processing unit 31 is constructed using a processing unit such as a CPU or MPU, ROM, and RAM. The processing unit 31 reads and executes the program 32a stored in the storage unit 32 to perform processing such as retrieving dynamic images accumulated in the dynamic image DB 12b of the server device 1 and displaying (playing) these dynamic images.

[0070] The storage unit 32 is constructed using, for example, a non-volatile storage element such as flash memory or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In this embodiment, the storage unit 32 stores the program 32a executed by the processing unit 31. In this embodiment, the program 32a is distributed via a remote server device, and the terminal device 3 obtains the program 32a through communication and stores it in the storage unit 32. However, the program 32a may also be written to the storage unit 32 during the manufacturing stage of the terminal device 3. For example, the program 32a may also be read by the terminal device 3 from a storage medium 98 such as a memory card or optical disc and stored in the storage unit 32. For example, the program 32a may also be read by a writing device from a storage medium 98 and written to the storage unit 32 of the terminal device 3. The program 32a may be provided in the form of distribution via a network or in the form of storage in the storage medium 98.

[0071] The communication unit 33 communicates with various devices via a network N, including mobile phone communication networks, wireless LANs, and the Internet. In this embodiment, the communication unit 33 communicates with the server device 1 via the network N. The communication unit 33 sends data provided by the processing unit 31 to other devices and provides data received from other devices to the processing unit 31.

[0072] The display unit 34 is configured using a liquid crystal display or the like, and displays various images and text based on the processing of the processing unit 31. The operation unit 35 receives user operations and notifies the processing unit 31 of the received operations. For example, the operation unit 35 receives user operations via input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 34. Furthermore, the operation unit 35 may also be an input device such as a mouse and keyboard, and these input devices may be configured to be detachable from the terminal device 3.

[0073] Furthermore, in this embodiment, the terminal device 3 reads and executes the program 32a stored in the storage unit 32 via the processing unit 31, thereby implementing the retrieval processing unit 31a and the display processing unit 31b, etc., as software functional units in the processing unit 31. Additionally, the program 32a can be a program specific to the information processing system of this embodiment, or it can be a general-purpose program such as an internet browser or web browser.

[0074] The retrieval processing unit 31a performs retrieval processing on a large number of animated images stored in the animated image database 12b of the server device 1. The retrieval processing unit 31a accepts various retrieval conditions input by the user and sends the received retrieval conditions to the server device 1. Upon receiving the retrieval conditions from the terminal device 3, the server device 1 extracts animated images that match the retrieval conditions from the animated image database 12b and sends a summary of the extracted animated images, etc., as retrieval results to the terminal device 3. The terminal device 3 displays the retrieval results received from the server device 1. Based on the retrieval results, the retrieval processing unit 31a accepts the user's selection of the animated image to be played and requests the server device 1 to send the selected animated image.

[0075] The display processing unit 31b performs display processing such as displaying the screen receiving the input search criteria, displaying information sent from the server device 1 as search results, and playing (displaying) animated images. Based on the animated image transmission request from the search processing unit 31a, the server device 1 reads the requested animated image from the animated image database 12b and sends the animated image along with various information related to it (e.g., scene change positions and keyframe information) to the requesting terminal device 3. The display processing unit 31b of the terminal device 3, which receives the animated image from the server device 1, plays the received animated image and displays it on the display unit 34. Furthermore, the display processing unit 31b receives the information sent along with the animated image and, based on the scene change positions and keyframe information included in the received information, can, for example, perform processing such as jumping the playback position of the animated image to a user-specified scene change position or keyframe.

[0076] <Detection and processing of scene changes and keyframes>

[0077] Figure 4 This is a schematic diagram illustrating the scene change location and keyframe detection processing performed by the information processing system of this embodiment. For example... Figure 4 As shown at the top, the moving images processed by the information processing system of this embodiment are, for example, formed from approximately several dozen consecutive frames (still images) within one second. Furthermore, the moving images can be segmented into multiple scenes. In this embodiment, a scene, such as a shot or scene in video production, is a unit for dividing the actions of people or objects presented in the moving image. Furthermore, in this embodiment, as... Figure 4 As shown in the lower part, a scene is obtained by taking multiple frames consecutively, including at least one keyframe, and the first and last frames are treated as the positions where the scene changes.

[0078] In this embodiment, the server device 1 of the information processing system acquires dynamic images of the construction or repair of the air conditioning equipment 101 captured by the camera 103 via a communication or storage medium, and stores the acquired dynamic images in the dynamic image DB 12b. At this time, the server device 1 performs scene change position and keyframe detection processing on the acquired dynamic images, and establishes a correspondence between the detected scene change position and keyframe and the dynamic images, storing this information in the dynamic image DB 12b.

[0079] (1) Detection of candidates for changing positions on the field

[0080] For the moving image acquired from camera 103, server device 1 first performs a process to detect candidate locations of scene changes included in the moving image. In this embodiment, server device 1 calculates the difference between two temporally consecutive frames among the multiple frames constituting the moving image, and determines whether the calculated value exceeds a predetermined threshold. If the difference between the frames exceeds the threshold, server device 1 considers the locations of the two consecutive frames as candidates for scene change locations. In this embodiment, server device 1 calculates the Bach distance of the HSL histogram and the distance of the hash value as the values ​​representing the frame differences.

[0081] Figure 5 This is a schematic diagram representing an example of an HSL histogram, which is a graph showing the distribution of the number of pixels in a sample frame relative to the values ​​of H (hue), S (saturation), and L (luminance). Figure 5 The horizontal axis of the chart represents the HSL value, and the vertical axis represents the number of pixels. Figure 5In the chart, the distribution of H is represented by a solid line, the distribution of S by a dashed line, and the distribution of L by a dotted line. Server device 1 calculates the HSL histogram for each frame constituting the dynamic image. Server device 1, for example, can convert the pixel values ​​of a frame provided as RGB values ​​into HSL values, and calculate the HSL histogram by counting the number of pixels contained in the frame for each HSL value.

[0082] Server device 1 calculates the HSL histogram for all frames included in the moving image, and calculates a value representing the difference between the HSL histogram and the temporally consecutive preceding frame for each frame. In this embodiment, server device 1 calculates the Bartholomew's distance as a value representing the difference in the HSL histograms. Furthermore, the method for calculating the Bartholomew's distance is existing technology, so detailed explanation is omitted. In addition, while server device 1 calculates the HSL histogram for each frame and calculates the Bartholomew's distance between frames in this embodiment, it is not limited to this; server device 1 may also calculate statistical values ​​other than the HSL histogram, or values ​​other than the Bartholomew's distance, as values ​​representing the difference between frames.

[0083] Furthermore, server device 1 calculates hash values ​​for all frames included in the dynamic image, such as calculating pHash. pHash is obtained by performing frequency transformation on the image through discrete cosine transform, extracting low-frequency components, and calculating hash values. Additionally, server device 1 calculates the Hamming distance of the pHash between each frame and the temporally consecutive preceding frame, as a value representing the difference between frames. pHash has the characteristic that the more similar two images are, the smaller the Hamming distance between their pHash values. Furthermore, the calculation methods for pHash and Hamming distance are existing technologies, so detailed descriptions are omitted. In this embodiment, server device 1 calculates the pHash for each frame and the Hamming distance of the pHash between frames, but it is not limited to this; server device 1 may also calculate hash values ​​other than pHash (e.g., aHash (average hash)) or values ​​other than Hamming distance as values ​​representing the difference between frames.

[0084] Thus, in this embodiment, the server device 1 calculates the Bartholomew's distance and the Hamming distance of the HSL histogram between each frame and the preceding frame that is sequentially consecutive in time, and uses these values ​​as values ​​representing the difference between the two frames. In this embodiment, the average value (weighted average) of the Bartholomew's distance and the Hamming distance calculated by the server device 1 is calculated, and the calculated average value is used as the inter-frame difference.

[0085] Figure 6 This is a schematic diagram illustrating an example of the calculation results of the inter-frame difference of a moving image, and a graph showing the change of the difference over the playback time of the moving image. Figure 6 The horizontal axis of the graph represents time (playback time of the moving image), and the vertical axis represents the difference between frames. Server device 1 calculates the difference between each frame and the previous frame for all frames included in the moving image and determines whether this difference exceeds a predetermined threshold. Figure 6 In the diagram, the solid lines represent the changes in the degree of difference, while the dashed horizontal lines represent the threshold.

[0086] Server device 1 selects moments when the calculated difference exceeds a predetermined threshold as candidates for the location of the scene change. Figure 6 In the image, the vertical line of the single-dotted line is considered by server device 1 as a candidate moment for a change in video position. Additionally, in... Figure 6 There are three candidate moments for scene change locations, and there are also moments where the difference exceeds a threshold. This is because, in this embodiment, after detecting a candidate moment for a scene change location, the server device 1 limits the detection of such candidates for a predetermined time. By setting such a limit, it is expected that a large number of similar scene change location candidates will be suppressed.

[0087] (2) Detection of candidate frames

[0088] Server device 1 performs a process to detect candidate frames that are potential keyframes from all frames included in the dynamic image captured by camera 103. Furthermore, the candidate frame detection process can be performed before, after, or simultaneously with the aforementioned scene change location detection process. In this embodiment, server device 1 performs edge extraction image processing on multiple frames constituting the dynamic image.

[0089] Figure 7 This is a schematic diagram illustrating an example of edge extraction. In Figure 7 The left side shows an example of an image (frame image) equivalent to a single frame included in a moving image. Figure 7 On the right side, regarding the edges extracted from the image of that frame, a binary image (edge ​​image) is shown, in which pixels corresponding to the edges are set to white and pixels outside the edges are set to black. Server device 1 can extract edges from the image of each frame by performing edge detection processing, for example, using the Carney method, on each frame included in the dynamic image. Image processing for extracting edges from an image is a prior art, therefore detailed explanation is omitted. Alternatively, the edge extraction from the frame image performed by server device 1 can also be performed using any image processing method.

[0090] Server device 1, which extracts edges from each frame of a dynamic image, compares the edges of two temporally consecutive frames and calculates the edge change rate. Server device 1 can compare two edge images extracted from two frames, for example, calculating the sum of the number of edge pixels becoming non-edge pixels and the number of non-edge pixels becoming edge pixels, and then calculating the ratio of this sum to the total number of pixels in a frame, using this ratio as the edge change rate. Furthermore, the edge change rate can be any value that represents the degree to which the edge changes between frames; the above-described method for calculating the edge change rate is just one example and is not limited to it. Server device 1 can also use any method to calculate the edge change rate.

[0091] Figure 8 This is a schematic diagram illustrating an example of the calculated results of the edge change rate between frames of a moving image, and a graph showing the change of the edge change rate over the playback time of the moving image. Figure 8 The horizontal axis of the graph represents time (playback time of the moving image), and the vertical axis represents the edge change rate between frames. Server device 1 compares all frames included in the moving image with the previous frame and calculates the edge change rate, then determines whether the edge change rate exceeds a predetermined threshold. Figure 8 In the diagram, the horizontal straight line represents the threshold.

[0092] Server device 1 selects frames whose calculated edge change rate is below a predetermined threshold (times less than the threshold) as candidate frames to become keyframes. Figure 8 In the diagram, the vertical straight line represents the moment when a frame is selected as a candidate frame by server device 1. Figure 8 The image shows that three candidate frames were detected in the first half of the video, and one candidate frame was detected in the second half. Additionally, in... Figure 8 In addition to these four candidate frames, there are also moments when the edge change rate is lower than the threshold. Server device 1 can also detect candidate frames for these moments.

[0093] (3) Determination of key frames

[0094] Server device 1 performs a process to determine the final keyframe by removing similar candidate frames from all frames included in the moving image, for example, when multiple candidate frames exist within a specified time. Furthermore, server device 1 performs the keyframe determination process after the candidate frame detection process, but it can also perform the keyframe determination process before, after, or in parallel with the detection of candidates for scene change positions.

[0095] The server device 1 in this embodiment performs, for example, the extraction of feature quantities from each candidate frame. It compares the feature quantities of two candidate frames and calculates their similarity. If the calculated similarity exceeds a threshold, the two candidate frames are determined to be similar. In this embodiment, the server device 1 extracts ORB keypoints as feature quantities for each candidate frame. ORB is a method that combines keypoint detection using FAST (Accelerated Segmentation Test Features) and feature quantity descriptors using BRIEF (Binary Robust Independent Basic Features). The aforementioned ORB, FAST, and BRIEF technologies are existing technologies, therefore, detailed descriptions are omitted. Furthermore, keypoint extraction performed by the server device 1 can also be performed using methods other than ORB, such as SIFT (Scale Invariant Feature Transform) or SURF (Accelerated Robust Features). Additionally, the server device 1 can also extract feature quantities other than keypoints from the frame. Figure 9 This is a schematic diagram illustrating an example of key points extracted from candidate frames. Figure 9 The two images shown were obtained by extracting key points from two similar candidate frames (candidate frame 1 and candidate frame 2). The extracted key points are indicated by circular dots on the images.

[0096] Next, server device 1 compares the key points extracted from the two candidate frames and performs a process to search for matching key points, also known as key point matching. Key point matching extracted using methods such as ORB is existing technology; therefore, detailed explanation is omitted in this embodiment. Figure 10 This is a schematic diagram illustrating an example of keypoint matching results. In Figure 10 In the example shown, by Figure 9 The keypoints in the two candidate frames shown are connected by straight lines, indicating the matching results.

[0097] For example, server device 1 counts the total number of keypoints extracted from two candidate frames and the number of keypoint matches between the two candidate frames, and calculates the ratio of the number of matches to the total number of keypoints as the similarity. Server device 1 can determine whether the calculated similarity exceeds a predetermined threshold, and if the similarity exceeds the threshold, determine that the two candidate frames are similar.

[0098] Server device 1 identifies candidate frames for which no similar candidate frames exist as keyframes in the motion picture. Furthermore, when the motion picture includes multiple similar candidate frames, server device 1 appropriately selects one candidate frame from these multiple candidate frames and identifies it as a keyframe, removing one or more unselected candidate frames from the keyframe candidate pool. In this case, if two similar candidate frames exist, server device 1 may, for example, select the earlier-timed candidate frame as the keyframe and remove the later-timed candidate frame from the keyframe candidate pool. Similarly, if three similar candidate frames exist, server device 1 may, for example, select the second-timed candidate frame as the keyframe and remove the first and third candidate frames from the keyframe candidate pool. Moreover, the method for selecting a candidate frame as a keyframe from multiple similar candidate frames is not limited to the methods described above; server device 1 can select a keyframe from multiple similar candidate frames using any method.

[0099] (4) Determining the location of scene changes

[0100] After detecting candidate locations for scene changes and identifying keyframes from the moving image, server device 1 performs a process to determine the scene change location from the candidate locations. In this embodiment, server device 1 determines the final scene change location from the candidate locations based on the temporal order relationship between the candidate locations and the keyframes, thereby determining the scene included in the moving image. In this embodiment, as... Figure 4 As shown in the lower part, the scene of the moving image includes at least one keyframe.

[0101] Figure 11 This is a schematic diagram illustrating a method for determining scene change positions in server device 1. The illustrated example shows server device 1 detecting the states of three candidate scenes 1-3 and two keyframes 1 and 2 by performing the aforementioned processing on a motion image. Temporally, the scene change position candidates are arranged in the order of candidate 1, candidate 2, and candidate 3, with keyframes 1 and 2 arranged in this order between candidates 2 and 3. There are no keyframes between candidates 1 and 2. The motion image consisting of multiple frames existing between candidates 1 and 2 is designated as scene candidate 1, and the motion image consisting of multiple frames (including keyframes 1 and 2) between candidates 2 and 3 is designated as scene candidate 2.

[0102] In this embodiment, the moving image includes one or more scenes, and each scene includes one or more keyframes. Figure 11The scene change position candidates 1 and 2 shown are scene change position candidates 1 for preceding and following scenes, but do not include keyframes and are not equivalent to the scenes in this embodiment. When a keyframe is not included between two temporally consecutive scene change position candidates, the server device 1 removes either of the two scene change position candidates and links the scene candidate without a keyframe to the scene candidate containing a keyframe. Figure 11 In the example shown, server device 1, for example, removes candidate 2 of the scene change position, thereby connecting the scene candidates 1 and 2 existing before and after candidate 2 of the scene change position into a single scene, and takes candidate 1 and 3 of the scene change position as the final scene change position.

[0103] Furthermore, in the above example, for scene candidate 1 that does not contain a keyframe, server device 1 removes candidate 2 at a later time-series scene change position and links scene candidate 1 with the later time-series scene candidate 2, but is not limited to this. Server device 1 may also remove candidate 1 at an earlier time-series scene change position and link scene candidate 1 with the scene candidate preceding it. For scene candidates that do not contain a keyframe, server device 1 may remove candidates at earlier time-series scene change positions or candidates at later time-series scene change positions. Which scene change position candidate to remove can be predetermined, for example, or selected based on the length of the preceding and following scene candidates or the number of keyframes.

[0104] Furthermore, server device 1 can also remove candidates for scene change locations if they exist between two temporally consecutive and similar keyframes. Server device 1 can use the similarity calculated when determining keyframes from candidate frames to determine whether two keyframes are similar. Server device 1 calculates the similarity based on keypoint matching results for two temporally consecutive keyframes, and determines that the two keyframes are similar if the calculated similarity exceeds a predetermined threshold (but is a value smaller than the threshold used to determine whether candidate frames are similar). Server device 1 can remove candidates for scene change locations existing between two similar keyframes and connect the scene candidates before and after the candidate for the scene change location to form a single scene.

[0105] Furthermore, server device 1 can also remove the scene between two scene change locations from the overall motion picture if the keyframe does not exist between them. Figure 11In the example shown, if the keyframe does not exist between candidates 1 and 2 at the scene change position, server device 1 can also presume that the motion picture between them does not contain important information, and remove scene candidate 1 between candidates 1 and 2 at the scene change position from the motion picture. Alternatively, server device 1 can remove either candidate 1 or candidate 2 at the scene change position along with scene candidate 1.

[0106] After determining all scene change locations and keyframes included in the moving image, server device 1 establishes a correspondence between the determined scene change locations and keyframes and the moving image, and stores this correspondence in the moving image DB 12b. Furthermore, when server device 1 receives a request from terminal device 3 to send the moving image stored in the moving image DB 12b, it reads the requested moving image and the corresponding information related to scene change locations and keyframes from the moving image DB 12b and sends it to the requesting terminal device 3. Terminal device 3 can use the information related to scene change locations and keyframes received from server device 1 along with the moving image, for example, to accept the user's scene selection and start playing and displaying the moving image from the received scene.

[0107] Figure 12 and Figure 13 This is a flowchart illustrating the processing steps performed by the server device 1 in this embodiment. The processing unit 11 of the server device 1 in this embodiment, for example, communicates with the camera 103 via the communication unit 13 to acquire a dynamic image of the construction or repair of the air conditioning equipment 101 captured by the camera 103 (step S1). The scene change position candidate detection unit 11a of the processing unit 11 calculates the HSL histogram of each frame included in the dynamic image acquired in step S1 (step S2). Based on the HSL histogram of each frame calculated in step S2, the scene change position candidate detection unit 11a calculates the Bartholomew's distance between the HSL histograms of two temporally consecutive frames (step S3).

[0108] Next, the scene change location candidate detection unit 11a calculates the pHash of each frame included in the dynamic image acquired in step S1 (step S4). Based on the pHash of each frame calculated in step S4, the scene change location candidate detection unit 11a calculates the Hamming distance between the pHash of two temporally consecutive frames (step S5). Based on the Bartholomew's distance calculated in step S3 and the Hamming distance calculated in step S5, the scene change location candidate detection unit 11a calculates the difference between two temporally consecutive frames (step S6). The difference can be, for example, the sum or average of the Bartholomew's distance and the Hamming distance. The scene change location candidate detection unit 11a compares the difference calculated in step S6 with a predetermined threshold, and detects two frames (or the frames between these two frames) with a difference exceeding the threshold as candidates for scene change locations (step S7).

[0109] Furthermore, the candidate frame detection unit 11b of the processing unit 11 extracts the edges of each frame included in the dynamic image acquired in step S1 (step S8). Based on the edges extracted in step S8, the candidate frame detection unit 11b calculates the rate of change of the edges between each frame and the previous frame in the time sequence (step S9). The candidate frame detection unit 11b compares the rate of change of the edges calculated in step S9 with a predetermined threshold, and detects frames whose rate of change of the edges does not exceed the threshold as candidate frames to become keyframes (step S10).

[0110] The keyframe determination unit 11c of the processing unit 11 extracts key points of the ORB for each candidate frame detected in step S10 (step S11). Based on the key points extracted in step S11, the keyframe determination unit 11c performs key point matching between candidate frames (step S12). Based on the key point matching results of step S12, the keyframe determination unit 11c calculates the similarity between candidate frames (step S13). For multiple candidate frames with similarity exceeding a threshold, the keyframe determination unit 11c selects one frame as the keyframe and removes the other candidate frames, thereby determining the keyframe (step S14).

[0111] The scene change position determination unit 11d of the processing unit 11, based on the scene change position candidates detected in step S7 and the keyframes determined in step S14, retrieves scene candidates that do not contain keyframes from the scene candidates existing between two temporally consecutive scene change position candidates (step S15). The scene change position determination unit 11d removes either the scene change position candidate before or after the scene change position candidate that does not contain keyframes retrieved in step S15 (step S16). The scene change position determination unit 11d determines the scene change position candidate that was not removed in step S16 as the final scene change position (step S17). The DB processing unit 11e of the processing unit 11 establishes a correspondence between the motion image acquired in step S1 and the information related to the scene change position determined in step S17 and the keyframe determined in step S14 and stores it in the motion image DB 12b (step S19), and ends the processing.

[0112] <Playback Processing of Dynamic Images>

[0113] In the information processing system of this embodiment, the server device 1 stores information related to the scene change positions and keyframes of the moving image determined by the above processing in the moving image DB 12b in a manner corresponding to the moving image. Furthermore, the moving image DB 12b stores various types of information in a corresponding manner, such as strings like the title or description of the moving image given by the photographer, the shooting date of the moving image, and the shooting location of the moving image. In addition, in this embodiment, the moving image DB 12b stores moving images obtained from filming the construction or repair of the air conditioning equipment 101, and stores textual information such as the name or product number of the air conditioning equipment 101, which is the object of the construction or repair work, in a corresponding manner.

[0114] Terminal device 3, for example, accepts a string input by the user as a keyword and sends the received string to server device 1 to request the retrieval of a moving image. Server device 1, based on the request from terminal device 3, retrieves moving images from moving image DB 12b that contain the provided keyword string, such as the title, description, name of air conditioning equipment 101, or product number, and sends information related to the matching moving images as search results to the requesting terminal device 3. Terminal device 3, having received the search results from server device 1, displays a list of information such as the title and shooting date of the moving images matching the input keyword.

[0115] Furthermore, terminal device 3 accepts the user's selection of a video to be played from the video images displayed as a list of search results, and requests server device 1 to send the selected video image. Based on the request from terminal device 3, server device 3 reads the selected video image and information related to scene change positions and keyframes corresponding to that video image from video image DB 12b and sends it to the requesting terminal device 3. Terminal device 3, having received the video image and related information to scene change positions and keyframes from server device 1, displays the video image playback screen on display unit 34, playing and displaying the video image on that screen.

[0116] Figure 14 This is a schematic diagram illustrating an example of the playback screen of terminal device 3. In the illustrated playback screen, a dynamic image display area for displaying dynamic images is located in the upper center of the screen, and four operation buttons are arranged horizontally below this area. From left to right, the four operation buttons are: a button for returning to the beginning of a scene (Back button), a button for playing the dynamic image (Play button), a button for skipping to the next scene (Skip button), and a button for stopping the playback of the dynamic image (Stop button). Terminal device 3 accepts user input to these operation buttons and performs actions such as playing and stopping the dynamic image. Furthermore, based on the scene change position received from server device 1, terminal device 3 performs scene switching of the dynamic image using the Back and Skip buttons. For example, if the Back button is pressed, terminal device 3 starts playback from the closest scene change position before the current playback time of the dynamic image. Similarly, if the Skip button is pressed, terminal device 3 starts playback from the closest scene change position after the current playback time of the dynamic image.

[0117] Furthermore, in the playback screen, the terminal device 3 displays a progress bar below the four operation buttons, indicating the playback time. The progress bar shows segments indicating the positions of scene changes in the moving image. The illustrated example shows a moving image comprising three scenes, with the progress bar showing two segments of thick vertical lines indicating video changes. Additionally, the terminal device 3 displays scaled-down images of one or more keyframes included in the moving image below the progress bar, configured appropriately. The terminal device 3 indicates the time when these keyframes appear in the moving image by connecting the keyframe images to the progress bar with arrows. These arrows indicate the position of the corresponding keyframe's appearance time within the playback time of the moving image shown on the progress bar. Upon receiving an operation such as a click or tap on a keyframe displayed in the playback screen, the terminal device 3 resumes playback of the moving image from the moment corresponding to the keyframe whose operation was received.

[0118] in addition, Figure 14 The method of displaying the dynamic image shown is an example and is not limited thereto. The terminal device 3 can display dynamic images in any way.

[0119] <Summary of Dynamic Image Generation and Processing>

[0120] In the information processing system of this embodiment, the server device 1 performs processing to generate a summary animated image (summary animated image) based on the scene change positions and keyframes of the animated image determined by the above processing. The server device 1 extracts (cuts out) one or more portions of the animated image from the animated image and links them together to generate a summary animated image with a shorter playback time than the original animated image.

[0121] In this embodiment, the server device 1 extracts a portion of the motion image from the overall motion image, spanning a predetermined time (e.g., a few seconds to tens of seconds) from the position of the scene change, and a portion of the motion image before and after a predetermined time before the keyframe. The server device 1 concatenates the aforementioned portion of the motion image extracted from the motion image in chronological order to generate a summary motion image. The server device 1 establishes a correspondence between the generated summary motion image and the original motion image and stores it in the motion image DB 12b.

[0122] Server device 1 can generate a summary animated image, for example, when acquiring animated images from camera 103. It can also generate a summary animated image based on a request from terminal device 3, or at other times. Furthermore, server device 1 can, for example, read a summary animated image of a matching animated image from animated image DB 12b as a search result and send it to terminal device 3 when sending search results in response to a search request for animated images from terminal device 3. Terminal device 3 can also display the summary animated image along with information such as the titles of multiple animated images matching the search criteria as a summary of the animated image search results.

[0123] <Title Generation and Processing>

[0124] In the information processing system of this embodiment, the server device 1 can use a learning model that has undergone machine learning, namely AI (artificial intelligence), to automatically generate titles for moving images, titles for scenes included in the moving images, or titles for keyframes included in the moving images. Figure 15 This is a schematic diagram used to explain the learning model used by the server device 1 in this embodiment. The learning model used by the server device 1 in this embodiment is a machine learning model that has been pre-processed to accept keyframes included in the dynamic image as input and output the category of construction or repair of the air conditioning equipment 101 presented in the keyframe as a classification result.

[0125] The learning model in this embodiment can, for example, employ a learning model with a CNN (Convolutional Neural Network) or DNN (Deep Neural Network) structure. The learning model can be generated, for example, by using image data corresponding to the construction or repair categories presented in the image, and performing so-called supervised machine learning. The supervised learning process for the learning model is existing technology, so detailed descriptions are omitted. However, the server device 1 can, for example, learn the model using methods such as gradient descent, probabilistic gradient descent, or error backpropagation.

[0126] Server device 1 stores the learning model generated by machine learning in storage unit 12. After detecting keyframes included in the dynamic image, server device 1 inputs one or more detected keyframes into the learning model and obtains the classification results output by the learning model. The category of construction or repair of air conditioning equipment 101 output by the learning model is, for example, construction of the outdoor unit of an air conditioner or repair of the indoor unit of an air conditioner. For example, server device 1 classifies the category of construction or repair using the learning model for each keyframe included in the dynamic image, obtains the classification results of all keyframes, and generates the title of the dynamic image and the title of the scene based on the obtained classification results.

[0127] Server device 1 appropriately selects a classification result from the classification results of multiple keyframes included in the moving image, such as selecting the most frequent classification result. For example, server device 1 can use a string such as "Year YY Month ZZ Day: Construction of outdoor unit of air conditioner" obtained by combining the selected classification result with the time the moving image was captured as the title of the moving image. Furthermore, server device 1 can appropriately select a classification result of keyframes included in each scene included in the moving image and use a string such as "Repair of indoor unit of air conditioner" corresponding to the selected classification result as the scene title. Moreover, the method implemented by server device 1 for generating titles based on the classification result of construction or repair categories is not limited to the above method; any method can be used.

[0128] Furthermore, as an alternative to a learning model that classifies the category of construction or repair of the air conditioning equipment 101 presented in the keyframe, the server device 1 can also use a learning model that accepts keyframe input and generates a string representing the title of that keyframe. Such a learning model can be constructed, for example, by combining a learning model such as a CNN that converts an image into feature quantities with a learning model such as an RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), BERT (Bidirectional Encoding Representation Based on Transformer), or GPT-3 (Generative Pre-trained Transformer-3) that generates a string representing the title based on those feature quantities.

[0129] Server device 1 establishes a correspondence between the title of the animated image generated based on keyframes and the titles of the scenes included in the animated image and stores them in the animated image DB 12b. Alternatively, for example, if the photographer or other personnel input the title of the animated image or scene, server device 1 can also store the input title in the animated image DB 12b without generating a title using a learning model. Furthermore, terminal device 3, for example, can... Figure 14 In the playback screen shown, the title of the moving image is displayed at the top, and the title of the scene is displayed in a way that corresponds to the image of one or more keyframes.

[0130] <Summary>

[0131] In the information processing system of this embodiment with the above structure, the server device 1 detects candidate locations of scene changes from the moving image captured by the camera 103, detects candidate frames that can become key frames (main frames) of the scene from multiple frames constituting the moving image, determines key frames from the candidate frames, and determines the scene change location from the candidate locations based on the temporal order of the candidate locations of the scene changes and the key frames. Therefore, the information processing system of this embodiment is expected to detect scene changes with high accuracy from the moving image.

[0132] Furthermore, in the information processing system of this embodiment, when a keyframe does not exist between two temporally sequential candidates for scene change positions, the server device 1 determines the scene change position by removing either of the two candidates. Additionally, in the information processing system of this embodiment, when a candidate for a scene change position exists between two temporally consecutive and similar keyframes, the server device 1 can also determine the scene change position by removing that candidate. Therefore, the information processing system of this embodiment is expected to determine the scene change position with high accuracy from the candidates for scene change positions detected in the moving image.

[0133] Furthermore, in the information processing system of this embodiment, the server device 1 calculates the statistical values ​​(HSL histogram) of each frame included in the dynamic image, and detects candidates for scene change locations based on the difference (difference degree) of the statistical values ​​of two consecutive frames in time. Additionally, in the information processing system of this embodiment, the server device 1 calculates the hash value (pHash) of each frame included in the dynamic image, and detects candidates for scene change locations based on the difference in the hash values ​​of two consecutive frames in time. Therefore, the information processing system of this embodiment is expected to detect candidates for scene change locations from dynamic images with high accuracy.

[0134] Furthermore, in the information processing system of this embodiment, the server device 1 extracts edges from each frame included in the dynamic image and detects candidate frames based on the change (rate of change) of edges between two consecutive frames in time sequence. In addition, in the information processing system of this embodiment, the server device 1 extracts feature points (key points) from the candidate frames and removes similar candidate frames based on the comparison results of feature points between multiple candidate frames, thereby determining key frames from the candidate frames. Therefore, the information processing system of this embodiment is expected to detect candidate frames with high accuracy from dynamic images and determine key frames with high accuracy from the candidate frames.

[0135] Furthermore, in the information processing system of this embodiment, the server device 1 establishes a correspondence between information related to the determined scene change position and keyframe and the moving image, and stores it in the moving image DB 12b. The terminal device 3 accepts the user's selection of the scene change position or keyframe, and plays the moving image based on the accepted scene change position or keyframe. Thus, the information processing system of this embodiment can expect to play moving images starting from the scene or the like desired by the user.

[0136] Furthermore, in the information processing system of this embodiment, the server device 1 establishes a correspondence between information related to the determined scene change location and keyframes and textual information such as the title, description, shooting date, shooting location, and name or product number of the air conditioning equipment (air conditioning-related equipment) 101 that is the object of construction or repair, and stores it in the dynamic image DB 12b. Thus, users can retrieve dynamic images by inputting keywords based on the textual information corresponding to the dynamic images.

[0137] Furthermore, in the information processing system of this embodiment, partial motion images are extracted from motion images based on determined scene change locations and keyframes, and the extracted partial motion images are combined to generate a summary motion image (summary motion image). Thus, the information processing of this embodiment can provide users with summary motion images, allowing users to easily grasp the summary of even long-running motion images.

[0138] Furthermore, in the information processing system of this embodiment, the server device 1 inputs the determined keyframes into a pre-processed machine learning model, and determines the title of the moving image or the scene included in the moving image based on the information output by the learning model. Thus, even if the photographer of the moving image does not input a title, the information processing system of this embodiment can automatically assign a title to the moving image.

[0139] It should be considered that the embodiments disclosed herein are illustrative in all respects and do not constitute a limitation. The scope of the invention is defined by the claims rather than by the foregoing, and is intended to include all changes within the meaning and scope equivalent to the claims.

[0140] Symbol Explanation

[0141] 1. Server device (image processing device);

[0142] 3. Terminal device;

[0143] 11. Processing Department;

[0144] 11a Candidate detection unit for scene change positions;

[0145] 11b Candidate Frame Detection Unit;

[0146] 11c Keyframe Determination Unit (Main Frame Determination Unit);

[0147] 11d Determination of the location of scene changes;

[0148] 11e DB Processing Unit;

[0149] 12. Storage Unit;

[0150] 12a Server Program (Computer Program);

[0151] 12b dynamic image database;

[0152] 13 Ministry of Communications;

[0153] 31 Processing Department

[0154] 31a Retrieval Processing Department;

[0155] 31b Display processing unit;

[0156] 32. Storage Unit;

[0157] 32a program;

[0158] 33 Ministry of Communications;

[0159] 34. Display Unit;

[0160] 35. Operations Department;

[0161] 101 Air conditioning equipment (air conditioning-related equipment);

[0162] 102. Workers;

[0163] 103 cameras;

[0164] N network.

Claims

1. An image processing method, characterized in that, The image processing device performs the following actions: Detecting candidates for locations of scene changes from dynamic images. Candidate frames that can become the main frames of the scene are detected from the frames that constitute the dynamic image. The primary frame is determined from the candidate frames. Based on the candidate locations of scene changes and the temporal order of the main frames, If there is no major frame between two candidate scene change locations arranged in time sequence, the scene change location is determined by removing either of the two scene change locations from the candidates. In cases where there are candidate locations for scene changes between two temporally similar main frames, the scene change location is determined by removing the candidate locations. This allows us to determine the position of the scene change from the candidates for the position of the scene change.

2. The image processing method according to claim 1, characterized in that, Calculate the statistical values ​​for each frame. Candidates for scene change locations are detected based on the difference in statistical values ​​between two frames.

3. The image processing method according to claim 1, characterized in that, Calculate the hash value of each frame. Candidates for scene change locations are detected based on the difference in hash values ​​between two frames.

4. The image processing method according to claim 1, characterized in that, Extract edges from each frame. Candidate frames are detected based on changes in the edges between two frames.

5. The image processing method according to claim 1, characterized in that, Extract feature points from candidate frames. Candidate frames are removed based on the comparison results of feature points between multiple candidate frames, thereby determining the main frame from the candidate frames.

6. The image processing method according to claim 1, characterized in that, Information related to the determined scene change locations and main frames is mapped to and stored in the motion image. Accept changes in scene location or selection of main frame. Playback of the dynamic images based on the selected scene change position or main frame.

7. The image processing method according to claim 6, characterized in that, The information associated with the determined scene change location and the main frame, as well as the text information associated with the dynamic image, will be mapped and stored.

8. The image processing method according to claim 7, characterized in that, To acquire dynamic images obtained from the construction or repair of air conditioning-related equipment. Determine the locations of scene changes and key frames from the acquired dynamic images. Establish and store correspondences between the determined scene change locations, information related to the main frames, and text information related to air conditioning equipment.

9. The image processing method according to claim 1, characterized in that, Based on the determined location of scene changes and key frames, a portion of the dynamic image is extracted from the dynamic image. A summary dynamic image is generated by combining the extracted partial dynamic images.

10. The image processing method according to claim 1, characterized in that, The identified key frames are input into a learning model, which classifies the construction or repair work based on the key frames of the dynamic images captured, showing construction or repair work involving air conditioning-related equipment. Obtain the classification results output by the learning model. The title of the moving image or the scene included in the moving image is determined based on the obtained classification results.

11. A computer program product, characterized in that, The computer program product causes the computer to perform the following processes: Detecting candidates for locations of scene changes from dynamic images. Candidate frames that can become the main frames of the scene are detected from the frames that constitute the dynamic image. The primary frame is determined from the candidate frames. Based on the candidate locations of scene changes and the temporal order of the main frames, If there is no major frame between two candidate scene change locations arranged in time sequence, the scene change location is determined by removing either of the two scene change locations from the candidates. In cases where there are candidate locations for scene changes between two temporally similar main frames, the scene change location is determined by removing the candidate locations. This allows us to determine the position of the scene change from the candidates for the position of the scene change.

12. An image processing apparatus, characterized in that, include: A scene change location candidate detection unit detects candidates for scene change locations from a dynamic image; The candidate frame detection unit detects candidate frames from the frames constituting the dynamic image that can become the main frames of the scene; A main frame determination unit determines a main frame from the candidate frames; as well as The scene change position determination unit determines the scene change position based on the temporal order of the candidates for scene change positions and the main frames. If there is no main frame between two candidates for scene change positions in the temporal order, the scene change position is determined by removing either of the two scene change positions from the candidates. If there are candidates for scene change positions between two main frames that are temporally ordered and similar, the scene change position is determined by removing the scene change position from the candidates. Thus, the scene change position is determined from the candidates for scene change positions.

Images