Data processing device and program

Abstract

Provided is a data processing device comprising: a moving image data acquisition unit that acquires moving image data in which an object is captured; and a data generation unit that converts time-series motion information of the object in the moving image data into a still image element, and generates still image data including the element of the still image obtained through the conversion. The data generation unit may generate the still image data in which the time-series movement information of the object in the moving image data is represented by at least one of a pixel value, a feature vector, and an edge.

Description

Data processing device and program

[0001] The present invention relates to a data processing device and a program.

[0002] Patent Document 1 describes a method for automatically generating training data that reproduces various swimming movements of fish using musical scores corresponding to existing music. [Prior Art Documents] [Patent Documents] [Patent Document 1] Japanese Patent No. 7152535 General disclosure

[0003] According to one embodiment of the present invention, there is provided a data processing device, the data processing device may include a video data acquisition unit that acquires video data of an object, and a data generation unit that converts information on time-series movement of the object in the video data into still image elements and generates still image data including the converted still image elements.

[0004] In the data processing device, the data generation unit may generate the single still image data consisting of the multiple parts by at least one of generating one part of the multiple parts constituting the single still image data from one frame of the multiple frames of the video data and generating one part of the multiple parts constituting the single still image data from multiple frames of the video data. In any of the data processing devices, the data generation unit may generate the multiple parts for the multiple frames of the video data by converting information on three-dimensional positions of the multiple objects included in the frames into pixel value information. In any of the data processing devices, the video data acquisition unit may acquire multiple pieces of video data captured in parallel by multiple cameras that capture the object from multiple different directions, and the data generation unit may generate the multiple parts for each frame of the multiple pieces of video data by converting information on three-dimensional positions of the multiple objects included in the frames into pixel value information. The data generation unit may generate the still image data representing information on the time-series movement of the object in the video data by at least one of pixel values, feature vectors, and edges.

[0005] In any of the data processing devices, the data generation unit may convert information on changes in the time series positions of the plurality of objects in the video data into the elements of the still image, and generate the still image data including the elements of the converted still image. The data generation unit may generate the still image data representing the information on changes in the time series positions of the plurality of objects in the video data by pixel values. The data generation unit may convert information on three-dimensional positions where objects exist for each frame into pixel value information. The data generation unit may generate the still image data representing the information on changes in the time series positions of the plurality of objects in the video data by feature vectors. The data generation unit may convert information on three-dimensional positions where objects exist for each frame into feature vector information. The data generation unit may generate the still image data representing the information on changes in the time series positions of the plurality of objects in the video data by edges. The data generation unit may convert information on three-dimensional positions where objects exist for each frame into edge information.

[0006] In any of the data processing devices described above, the data generation unit may convert information on time-series shape changes of the object in the video data into the elements of the still image, and generate the still image data including the elements of the converted still image. The data generation unit may generate the still image data representing the information on time-series shape changes of the object in the video data by pixel values. The data generation unit may convert information on three-dimensional positions where multiple structural points of the object are located for each frame into pixel value information. The data generation unit may generate the still image data representing the information on time-series shape changes of the object in the video data by feature vectors. The data generation unit may convert information on three-dimensional positions where multiple structural points of the object are located for each frame into feature vector information. The data generation unit may generate the still image data representing the information on time-series shape changes of the object in the video data by edges. The data generation unit may convert information on three-dimensional positions where multiple structural points of the object are located for each frame into edge information.

[0007] In any of the data processing devices, the data generation section may convert information on the time-series movement of the object in the video data into text elements, and generate text data including the converted text elements.

[0008] In any of the data processing devices, the data generation unit may convert information on the time-series movement of the object in the video data into genome elements, and generate genome data including the converted genome elements.

[0009] Any of the data processing devices may further include a classification result acquisition unit that inputs the still image data generated by the data generation unit into a still image classification network that has been trained to classify still image data and acquires the classification result output from the still image classification network, and an alert output unit that outputs an alert when the classification result acquired by the classification result acquisition unit satisfies predetermined conditions.

[0010] Any of the data processing devices may further include a conversion data acquisition unit that acquires still image data including still image elements converted from information on the time-series movement of objects in the video data, and a video data generation unit that generates the video data from the still image data.

[0011] According to one embodiment of the present invention, there is provided a data processing device. The data processing device may include a still image data acquisition unit that acquires still image data including still image elements converted from information on the time-series movement of an object in video data. The data processing device may also include a video data generation unit that generates the video data from the still image data.

[0012] According to one embodiment of the present invention, there is provided a data processing device. The data processing device may include a video data acquisition unit that acquires video data capturing an object. The data processing device may include a data generation unit that converts information on the time-series movement of the object in the video data into text elements and generates text data including the converted text elements. The data generation unit may generate the text data representing the information on the time-series movement of the object in the video data using at least one of characters, words, fonts, and ASCII codes. The data generation unit may convert information on time-series positional changes of multiple objects in the video data into text elements and generate the text data including the converted text elements. The data generation unit may generate the text data representing the information on time-series positional changes of multiple objects in the video data using characters. The data generation unit may convert information on three-dimensional positions where objects exist for each frame into character information. The data generation unit may generate the text data representing the information on time-series positional changes of multiple objects in the video data using words. The data generation unit may convert information on the three-dimensional position where an object is located for each frame into word information. The data generation unit may generate the text data representing information on time-series positional changes of multiple objects in the video data using ASCII code. The data generation unit may convert information on the three-dimensional position where an object is located for each frame into ASCII code information. The data generation unit may generate the text data representing information on time-series positional changes of multiple objects in the video data using at least one of characters and words and a font. The data generation unit may convert information on the three-dimensional position where an object is located for each frame into at least one of characters and words and a font information. The data generation unit may convert information on time-series shape changes of objects in the video data into text elements, and generate the text data including the converted text elements.The data generation unit may generate the text data representing information on time-series shape changes of an object in the video data using characters. The data generation unit may generate the text data representing information on time-series shape changes of an object in the video data using ASCII code. The data generation unit may generate the text data representing information on time-series shape changes of an object in the video data using at least one of characters and words and a font. The data generation unit may generate the text data from a plurality of video data in which an object is captured from a plurality of different directions. The data generation unit may convert information on time-series movement of an object in the plurality of video data into text elements and generate the text data including the converted text elements. The data processing device may include a classification result acquisition unit that inputs the text data generated by the data generation unit to a still image classification network and acquires a classification result output from the still image classification network. The data processing device may include an alert output unit that outputs an alert according to the classification result obtained by the classification result acquisition unit.

[0013] According to one embodiment of the present invention, there is provided a data processing device. The data processing device may include a video data acquisition unit that acquires video data obtained by capturing an object. The data processing device may include a data generation unit that converts information on the time-series movement of the object in the video data into genome elements and generates genome data including the converted genome elements. The data generation unit may generate the genome data representing information on time-series positional changes of multiple objects in the video data using a base sequence. The data generation unit may convert information on the three-dimensional position at which an object exists for each frame into base sequence information. The data generation unit may convert information on time-series shape changes of an object in the video data into genome elements and generate the genome data including the converted genome elements. The data generation unit may generate the genome data representing information on time-series shape changes of an object in the video data using a base sequence. The data generation unit may convert information on the three-dimensional positions at which multiple structural points of an object exist for each frame into base sequence information. The data generation unit may generate the genome data from multiple video data obtained by capturing images of an object from multiple different directions. The data generation unit may convert information on the time-series movement of an object in the plurality of video data into genome elements and generate the genome data including the converted genome elements. The data generation unit may generate the genome data representing information on the time-series movement of an object in the plurality of video data by a base sequence. The data processing device may include a classification result acquisition unit that inputs the genome data generated by the data generation unit to a genome classification network and acquires the classification result output from the genome classification network. The data processing device may include an alert output unit that outputs an alert according to the classification result by the classification result acquisition unit.

[0014] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the data processing device.

[0015] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions.

[0016] 1 is an explanatory diagram illustrating an example of a data processing device 100. FIG. 2 is an explanatory diagram illustrating a conversion process for converting video data 200 into still image data 300. FIG. 3 is an explanatory diagram illustrating a conversion process for converting a plurality of video data 200 into still image data 300. FIG. 4 is an explanatory diagram illustrating a conversion process for converting a plurality of video data 200 into still image data 300. FIG. 5 is an explanatory diagram illustrating a conversion process for converting video data 200 into text data 400. FIG. 6 is an explanatory diagram illustrating a conversion process for converting video data 200 into text data 400. FIG. 7 is an explanatory diagram illustrating a conversion process for converting video data 200 into genome data 500. FIG. 8 is an explanatory diagram illustrating an example of the functional configuration of the data processing device 100. FIG. 9 is an explanatory diagram illustrating an example of the hardware configuration of a computer 1200 that functions as the data processing device 100.

[0017] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0018] In fish farming, understanding the condition of fish is extremely important to prevent waste of feed, mass mortality, disease, and so on. However, it is difficult to monitor a fish farm around the clock, and a monitoring system that automatically analyzes video footage of the fish farm and provides analysis results is desired. Such monitoring systems are needed for any object, not just fish. Such monitoring systems must analyze video data containing four-dimensional information consisting of three-dimensional space and time. However, analyzing video data directly imposes a very high processing load. In contrast, the data processing device 100 according to this embodiment converts video data into data with lower dimensions. For example, the data processing device 100 converts video data into two-dimensional data.

[0019] As a specific example, the data processing device 100 converts video data into still image data. The data processing device 100 may convert information contained in the video data into still image elements to convert the video data into still image data. For example, the data processing device 100 may perform object detection on the video data and convert information on the time-series movement of the object into still image elements to convert the video data into still image data. The data processing device 100 may convert the video data into still image data according to conversion information including a conversion rule. The data processing device 100 may realize object monitoring by analyzing the converted still image data. This reduces the processing load. Furthermore, neural networks for analyzing still image data have been extensively studied, and many neural networks already exist, making it possible to reuse such neural networks.

[0020] As a specific example, the data processing device 100 converts video data into text data. The data processing device 100 may convert information contained in the video data into text elements to convert the video data into text data. For example, the data processing device 100 may convert the video data into text data by performing object detection on the video data and converting information about the time-series movement of the object into text elements. The data processing device 100 may convert the video data into text data according to conversion information including conversion rules. The data processing device 100 may realize object monitoring by analyzing the converted text data. This can reduce the processing load. Furthermore, neural networks for analyzing text data have been studied very extensively, and many neural networks already exist, making it possible to reuse such neural networks.

[0021] The data processing device 100 may convert video data into data consisting of a list of specific characters or data consisting of a list of numbers. The data processing device 100 may, for example, convert video data into genome data. Genomic data may be data composed of base sequences. The data processing device 100 may convert information contained in the video data into genome elements, thereby converting the video data into genome data. For example, the data processing device 100 may convert the video data into genome data by performing object detection on the video data and converting information on the time-series movement of the object into genome elements. The data processing device 100 may convert the video data into genome data according to conversion information including conversion rules. The data processing device 100 may realize object monitoring by analyzing the converted genome data. This reduces the processing load. Furthermore, neural networks for analyzing genome data have been extensively studied, and many neural networks already exist, making it possible to reuse such neural networks.

[0022] 1 schematically illustrates an example of a data processing device 100. The data processing device 100 may acquire video data of an object and convert the acquired video data into data with a lower dimension than the video data (the converted data may be referred to as converted data). Examples of data with a lower dimension than video data include still image data, text data, and genome data.

[0023] The object may be anything that changes over time. The object may be one whose position changes over time, one whose shape changes over time, or one whose position and shape change over time. The object may be one whose position changes over time but whose shape does not change over time. The object may be one whose position changes over time but whose shape changes over time but whose position does not change over time.

[0024] The object may be a target of monitoring. In this embodiment, a case where the object is a fish and the target of monitoring is a school of fish will be mainly described as an example, but the present invention is not limited to this.

[0025] The data processing device 100 may acquire video data captured by the camera 102. The camera 102 may be built into the data processing device 100. The camera 102 may be externally attached to the data processing device 100. The camera 102 may be connected to the data processing device 100 via any network.

[0026] The data processing device 100 may receive video data from the communication terminal 30. The communication terminal 30 transmits video data captured by, for example, a camera 32 to the data processing device 100. The camera 32 may be built into the communication terminal 30. The camera 32 may be externally attached to the communication terminal 30. The camera 32 may be connected to the communication terminal 30 via any network.

[0027] The communication terminal 30 may be a PC (Personal Computer), a tablet terminal, a smartphone, or the like. The data processing device 100 and the communication terminal 30 may communicate via a network 20. The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: 5G (5th Generation) communication method, LTE (Long Term Evolution) communication method, 3G (3rd Generation) communication method, and 6G (6th Generation) communication method or later.

[0028] The data processing device 100 may pre-store a classification network trained to classify the transformed data. The data processing device 100 may input the transformed data to the classification network and obtain a classification result output from the classification network. The data processing device 100 may output an alert when the classification result satisfies a predetermined condition.

[0029] As a specific example, the data processing device 100 acquires video data of a school of fish in a fish farm, converts information on the time-series movements of multiple fish in the video data into still image elements, and generates still image data including the converted still image elements. The data processing device 100, for example, converts information on time-series changes in the positions of multiple fish in the video data into still image elements, and generates still image data including the converted still image elements. The data processing device 100 may generate still image data for any time unit, such as seconds, minutes, hours, days, weeks, months, or years.

[0030] For example, the data processing device 100 generates still image data in one-hour increments. In this case, one still image data includes characteristics of time-series changes in the positions of multiple fish over one hour. For example, five still image data are generated from five hours of video data. If multiple fish swim normally without any particular abnormalities over the five hours, the five still image data will be similar to each other. On the other hand, if multiple fish swim normally for four hours but then exhibit a more drastic movement in the last hour than before, the four still image data corresponding to the first four hours will be similar to each other, but the one still image data corresponding to the last hour may exhibit characteristics different from the four still image data. In other words, by classifying the still image data, changes in the state of the school of fish can be monitored.

[0031] The data processing device 100 may generate a still image classification network that classifies still image data by performing machine learning using the generated large amount of still image data. The data processing device 100 may input newly generated still image data into the generated still image classification network, thereby classifying the still image data.

[0032] The data processing device 100 may use an existing still image classification network. For example, the data processing device 100 may acquire and store in advance a still image classification network generated by another device. The data processing device 100 then inputs subsequently generated still image data into the still image classification network, thereby classifying the still image data.

[0033] The data processing device 100 may further include a function for generating video data from the converted data. The data processing device 100 may generate video data from the converted data that it has generated. The data processing device 100 may convert the converted data into video data by using the conversion information that was used when converting the video data into the converted data.

[0034] The data processing device 100 may generate video data from converted data generated by another device. The data processing device 100 may receive conversion information used to convert video data into converted data from the other device that generated the converted data, and may use the conversion information to convert the converted data received from the other device into video data.

[0035] The data processing device 100 may have a function of generating video data from converted data, without having a function of generating converted data from video data.

[0036] 2 is an explanatory diagram for explaining the conversion process for converting video data 200 into still image data 300. Here, a case will be explained in which the camera 102 converts video data 200, which captures an image of a school of fish in a fish farm 40, into still image data 300.

[0037] The data processing device 100 acquires video data 200 from the camera 102. The data processing device 100 converts information on the time-series movement of the school of fish in the video data 200 into still image elements, and generates still image data 300 including the converted still image elements.

[0038] For example, the data processing device 100 generates one portion 302 constituting the still image data 300 from one frame of the video data 200. The data processing device 100 may generate still image data 300 consisting of multiple portions 302 by generating multiple portions 302, each corresponding to a respective one of the multiple frames of the video data 200. Note that the data processing device 100 may also generate one portion 302 from multiple frames of the video data 200.

[0039] The data processing device 100 may generate the still image data 300 for each arbitrary period. For example, when generating the still image data 300 for each day, the data processing device 100 generates the still image data 300 corresponding to one day by generating a plurality of portions 302 from frames of one day's worth of video data 200.

[0040] The data processing device 100 may generate still image data 300 from a plurality of pieces of video data 200 captured in parallel by a plurality of cameras.

[0041] 3 and 4 are explanatory diagrams for explaining the conversion process of converting a plurality of video data 200 into still image data 300. Here, a case will be explained in which six video data 200, which are images of a school of fish in a fish farm 40 taken from six directions, are converted into still image data 300. In the example shown in FIGS. 3 and 4, the school of fish in the fish farm 40 are imaged from four directions, namely, above, below, and from the side (side A, side B, side C, side D).

[0042] Figure 3 shows examples of frame 211 of video data 210 captured from above, frame 221 of video data 220 captured from below, frame 231 of video data 230 captured from side A, frame 241 of video data 240 captured from side B, frame 251 of video data 250 captured from side C, and frame 261 of video data 260 captured from side D at a certain timing.

[0043] For example, the data processing device 100 generates portion 311 from frame 211, portion 321 from frame 221, portion 331 from frame 231, portion 341 from frame 241, portion 351 from frame 251, and portion 361 from frame 261. The data processing device 100 concatenates these as shown in Fig. 4. The data processing device 100 similarly generates and concatenates still image portions for the next frame. By repeating this process, the data processing device 100 generates still image data 300.

[0044] Note that the data processing device 100 may generate one portion from multiple consecutive frames, rather than generating one portion from one frame as described above. For example, the data processing device 100 generates one portion 311 from 10 consecutive frames of the video data 210 captured from above, generates one portion 321 from 10 consecutive frames of the video data 220 captured from below, generates one portion 331 from 10 consecutive frames of the video data 230 captured from side A, generates one portion 341 from 10 consecutive frames of the video data 240 captured from side B, generates one portion 351 from 10 consecutive frames of the video data 250 captured from side C, and generates one portion 361 from 10 consecutive frames of the video data 260 captured from side D. Ten frames is an example, and other numbers of frames may be used.

[0045] 5 is an explanatory diagram for explaining the conversion process for converting video data 200 into text data 400. Here, a case will be described in which video data 200, which is an image of a school of fish in a fish farm 40 captured by a camera 102, is converted into text data 400.

[0046] The data processing device 100 acquires video data 200 from the camera 102. The data processing device 100 converts information on the time-series movement of the school of fish in the video data 200 into text elements, and generates text data 400 including the converted text elements.

[0047] For example, the data processing device 100 generates one portion 402 constituting the text data 400 from one frame of the video data 200. The data processing device 100 may generate text data 400 consisting of multiple portions 402 by generating multiple portions 402, each corresponding to a respective one of the multiple frames of the video data 200. Note that the data processing device 100 may also generate one portion 402 from multiple frames of the video data 200.

[0048] The data processing device 100 may generate the text data 400 for each arbitrary period. For example, when generating the text data 400 for each day, the data processing device 100 generates the text data 400 corresponding to one day by generating a plurality of portions 402 from frames of the video data 200 for one day.

[0049] The data processing device 100 may generate text data 400 from multiple pieces of video data 200 captured in parallel by multiple cameras. Here, a case will be described in which six pieces of video data 200, which capture images of a school of fish in a fish farm 40 from six directions, are converted into text data 400, as shown in Figure 3.

[0050] For example, the data processing device 100 generates part 411 from frame 211, part 421 from frame 221, part 431 from frame 231, part 441 from frame 241, part 451 from frame 251, and part 461 from frame 261. The data processing device 100 concatenates these as shown in Fig. 6. The data processing device 100 similarly generates and concatenates text parts for the next frame. By repeating this process, the data processing device 100 generates text data 400.

[0051] Note that the data processing device 100 may generate one portion from multiple consecutive frames, rather than generating one portion from one frame as described above. For example, the data processing device 100 generates one portion 411 from 10 consecutive frames of the video data 210 captured from above, generates one portion 421 from 10 consecutive frames of the video data 220 captured from below, generates one portion 431 from 10 consecutive frames of the video data 230 captured from side A, generates one portion 441 from 10 consecutive frames of the video data 240 captured from side B, generates one portion 451 from 10 consecutive frames of the video data 250 captured from side C, and generates one portion 461 from 10 consecutive frames of the video data 260 captured from side D. Ten frames is an example, and other numbers of frames may be used.

[0052] 7 is an explanatory diagram for explaining the conversion process for converting video data 200 into genome data 500. Here, a case will be explained in which video data 200, which is an image of a school of fish in a fish farm 40 captured by a camera 102, is converted into genome data 500.

[0053] The data processing device 100 acquires video data 200 from the camera 102. The data processing device 100 converts information on the time-series movement of the school of fish in the video data 200 into genome elements, and generates genome data 500 including the converted genome elements.

[0054] For example, the data processing device 100 generates one portion constituting the genome data 500 from one frame of the video data 200. The data processing device 100 may generate genome data 500 consisting of multiple portions by generating multiple portions, each corresponding to a respective one of the multiple frames of the video data 200. Note that the data processing device 100 may also generate one portion 402 from multiple frames of the video data 200.

[0055] The data processing device 100 may generate the genome data 500 for any period of time. For example, when generating the genome data 500 for each day, the data processing device 100 generates the genome data 500 corresponding to one day by generating multiple parts from frames of one day's worth of video data 200.

[0056] The data processing device 100 may generate genome data 500 from multiple pieces of video data 200 captured in parallel by multiple cameras, similar to the still image data 300 and text data 400.

[0057] 8 schematically illustrates an example of the functional configuration of the data processing device 100. The data processing device 100 includes a conversion information storage unit 112, a model storage unit 114, a video data acquisition unit 116, a data generation unit 118, a classification result acquisition unit 120, a model generation unit 122, an output unit 130, a conversion data acquisition unit 142, and a video data generation unit 144.

[0058] The conversion information storage unit 112 stores conversion information including a conversion rule for converting the video data 200. The conversion information storage unit 112 may store conversion information that has been registered in advance. For example, the conversion information storage unit 112 stores conversion information including a conversion rule for converting the video data 200 into still image data 300. For example, the conversion information storage unit 112 stores conversion information including a conversion rule for converting the video data 200 into text data 400. For example, the conversion information storage unit 112 stores conversion information including a conversion rule for converting the video data 200 into genome data 500.

[0059] The model storage unit 114 stores a classification network trained to classify transformed data. The model storage unit 114 may store a pre-registered classification network. The model storage unit 114 stores, for example, a classification network generated by another device. For example, the transformation information storage unit 112 stores a still image classification network. For example, the transformation information storage unit 112 stores a text classification network. For example, the transformation information storage unit 112 stores a genome classification network.

[0060] The video data acquisition unit 116 acquires video data 200 obtained by capturing an image of an object. The video data acquisition unit 116 acquires the video data 200 to be converted. The video data acquisition unit 116 may receive the video data 200 from the camera 102. The video data acquisition unit 116 may receive the video data 200 from the communication terminal 30. The video data acquisition unit 116 may receive the video data 200 from another device. The video data acquisition unit 116 may acquire the video data 200 from a portable storage device or the like.

[0061] The video data 200 may be accompanied by a detection result of an object. The video data 200 may be accompanied by information on the three-dimensional position of the object. The information on the three-dimensional position of the object may be identified by a method using the output of a distance measuring sensor arranged separately from the camera 102 or the camera 32, a method using the image analysis results of the camera 102 or the camera 32, or the like. Note that the present invention is not limited to these, and any method may be used as long as it can identify the three-dimensional position of the object.

[0062] The data generation unit 118 generates converted data by converting the video data 200 acquired by the video data acquisition unit 116. The data generation unit 118 may generate the converted data from the video data 200 using the conversion information stored in the conversion information storage unit 112.

[0063] For example, the data generation unit 118 generates still image data 300 from the video data 200. The data generation unit 118 may convert information about the time-series movement of an object in the video data 200 into still image elements, and generate still image data 300 including the converted still image elements. The data generation unit 118 may generate still image data 300 that represents information about the time-series movement of an object in the video data 200 using at least one of pixel values, feature vectors, and edges. In this way, examples of still image elements include, but are not limited to, pixel values, feature vectors, and edges.

[0064] The data generating section 118 may convert information on time-series positional changes of multiple objects in the video data 200 into still image elements, and generate still image data 300 including the converted still image elements.

[0065] The data generation unit 118 generates still image data 300 that uses pixel values ​​to represent, for example, information about changes in the positions of multiple objects over time in the video data 200. For example, the data generation unit 118 converts, for each frame, information about the three-dimensional positions of the objects into pixel value information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple objects included in the frame as pixel values ​​for three consecutive pixels. As a result, for the portion of the still image data 300 corresponding to that frame, the pixel values ​​of the three consecutive pixels indicate the X, Y, and Z coordinates of the object. The data generation unit 118 can generate still image data 300 corresponding to the video data 200 by similarly assigning pixel values ​​for other frames of the video data 200. The method for converting the X, Y, and Z coordinate information of each of the multiple objects into pixel values ​​is not limited to this. Existing compression technology may be used to compress the X, Y, and Z coordinate information of each of the multiple objects and convert it into pixel values. The frame boundaries may be expressed by the layout, or one or more pixel values ​​may be used to indicate the frame boundaries. Other methods may also be used to indicate the frame boundaries.

[0066] Grayscale may be used as pixel values. This results in the generation of grayscale still image data 300 that represents information on changes in the positions of multiple objects over time in the video data 200. If multiple objects in the video data 200 are moving in a similar manner, the still image data 300 may have an appearance in which the same pattern is repeated. Furthermore, if multiple objects in the video data 200 are moving rapidly for a certain period of time, the still image data 300 may have an appearance in which that portion becomes darker, lighter, or changes more drastically, thereby indicating that that portion is different from the other portions.

[0067] Color may be used as the pixel value. This allows for more efficient representation of information about changes in the positions of multiple objects over time compared to when grayscale is used. When color is used, color still image data 300 is generated that represents information about changes in the positions of multiple objects over time in the video data 200. When multiple objects in the video data 200 are moving in a similar manner, the still image data 300 may have similar colors. Furthermore, when multiple objects in the video data 200 are temporarily moving vigorously, the still image data 300 may have a darker or lighter color in that portion, or the color may change drastically, thereby making that portion appear different from the other portions.

[0068] The data generation unit 118 generates still image data 300 that represents, for example, information on time-series positional changes of multiple objects in the video data 200 using feature vectors. The data generation unit 118 converts, for example, information on the three-dimensional positions of objects for each frame into feature vector information. The data generation unit 118 may use vectors that quantify position coordinates, velocity change amounts, and directions as a method for representing feature vectors, or may use a method for learning the features themselves. In the case of a large number of target objects, such as a school of fish, feature vectors may be generated after taking statistics on the time-series positions of multiple objects. Furthermore, statistics may be taken from the feature vectors and used as feature vectors in a higher layer.

[0069] The data generation unit 118 generates still image data 300 that uses edges to represent, for example, information about changes in the positions of multiple objects over time in the video data 200. The data generation unit 118 converts, for example, information about the three-dimensional positions of objects for each frame into edge information. The data generation unit 118 may realize the conversion between information about changes in the positions of multiple objects over time and edge information using, for example, a method of binarizing and differentiating, the Canny algorithm, or a method of learning edges or silhouettes using a deep neural network (DNN).

[0070] The data generating unit 118 may convert information on time-series shape changes of objects in the video data 200 into still image elements, and generate still image data 300 including the converted still image elements.

[0071] The data generation unit 118 generates still image data 300 that uses pixel values ​​to represent, for example, information on time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information on the three-dimensional positions of multiple structural points of an object for each frame into pixel value information. For example, if the object is a living thing or a robot, the structural points of the object may be the positions of its joints. If the object does not have joints, the structural points of the object may be each point on the object that affects the object's movement.

[0072] As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of multiple structural points of an object included in the frame as pixel values ​​for three consecutive pixels. As a result, for a portion of the still image data 300 corresponding to that frame, the pixel values ​​of three consecutive pixels indicate the X, Y, and Z coordinates of the object. The data generation unit 118 can generate still image data 300 corresponding to the video data 200 by similarly assigning pixel values ​​for other frames of the video data 200. The method for converting the X, Y, and Z coordinate information of each of multiple objects into pixel values ​​is not limited to this. Existing compression technology may be used to compress the X, Y, and Z coordinate information of each of multiple objects and convert it into pixel values. Frame divisions may be represented by arrangement, or one or more pixel values ​​may be used to indicate frame divisions. Methods other than these may also be used to represent frame divisions.

[0073] Grayscale may be used as pixel values. This results in the generation of grayscale still image data 300 that represents information on time-series shape changes of an object in the video data 200. If an object in the video data 200 moves in a similar manner, the still image data 300 may have an appearance in which the same pattern is repeated. Furthermore, if an object in the video data 200 moves vigorously for a certain period of time, the still image data 300 may have an appearance in which that portion becomes darker, lighter, or changes drastically, thereby indicating that that portion is different from the other portions.

[0074] Color may be used as the pixel value. This allows for more efficient representation of information about changes in the shape of an object over time than when grayscale is used. When color is used, color still image data 300 is generated that represents information about changes in the shape of an object over time in the video data 200. If objects in the video data 200 are moving in a similar manner, the still image data 300 may have a similar color tone. Furthermore, if an object in the video data 200 is moving rapidly for a certain period of time, the still image data 300 may have a darker or lighter color in that portion, or the color tone may change drastically, thereby making that portion appear different from the other portions.

[0075] The data generation unit 118 generates still image data 300 that uses feature vectors to represent, for example, information on time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information on the three-dimensional position where an object exists for each frame into feature vector information.

[0076] The data generation unit 118 generates still image data 300 that uses edges to represent, for example, information on time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information on the three-dimensional position where an object exists for each frame into edge information.

[0077] For example, the data generation unit 118 generates still image data 300 from a plurality of video data 200 captured from a plurality of different directions of an object. Here, the differences from generating still image data 300 from a single video data 200 will be mainly described. The data generation unit 118 may convert information on the time-series movement of an object in the plurality of video data 200 into still image elements, and generate still image data 300 including the converted still image elements. The data generation unit 118 may generate still image data 300 that represents information on the time-series movement of an object in the plurality of video data 200 by at least one of pixel values, feature vectors, and edges.

[0078] The data generating section 118 may convert information on time-series positional changes of multiple objects in multiple video data 200 into still image elements, and generate still image data 300 including the converted still image elements.

[0079] The data generation unit 118 generates still image data 300 that uses pixel values ​​to represent, for example, information about changes in the positions of multiple objects over time in multiple video data 200. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information about the three-dimensional positions of the objects present into pixel value information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X coordinate, Y coordinate, and Z coordinate of each of the multiple objects included in the frame as pixel values ​​for three consecutive pixels. As a result, for the portion of the still image data 300 corresponding to that frame, the pixel values ​​of three consecutive pixels indicate the X coordinate, Y coordinate, and Z coordinate of the object. The data generation unit 118 can generate still image data 300 corresponding to the video data 200 by similarly assigning pixel values ​​for other frames of the video data 200.

[0080] The data generation unit 118 generates still image data 300 that uses feature vectors to represent, for example, information on time-series positional changes of multiple objects in multiple video data 200. For example, the data generation unit 118 converts, for each of the multiple video data 200, information on the three-dimensional position where an object exists for each frame into feature vector information.

[0081] The data generation unit 118 generates still image data 300 that uses edges to represent, for example, information on time-series positional changes of multiple objects in multiple video data 200. For example, the data generation unit 118 converts, for each of the multiple video data 200, information on the three-dimensional position where an object exists for each frame into edge information.

[0082] The data generating unit 118 may convert information on time-series shape changes of objects in multiple video data 200 into still image elements, and generate still image data 300 including the converted still image elements.

[0083] The data generation unit 118 generates still image data 300 that uses pixel values ​​to represent, for example, information about time-series shape changes of an object in multiple video data 200. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information about the three-dimensional positions of multiple structural points of the object into pixel value information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X coordinate, Y coordinate, and Z coordinate of each of the multiple structural points of the object included in the frame as pixel values ​​for three consecutive pixels. As a result, for the portion of the still image data 300 corresponding to that frame, the pixel values ​​of three consecutive pixels indicate the X coordinate, Y coordinate, and Z coordinate of the object. The data generation unit 118 can generate still image data 300 corresponding to the video data 200 by similarly assigning pixel values ​​for other frames of the video data 200.

[0084] The data generation unit 118 generates still image data 300 that uses feature vectors to represent, for example, information on time-series shape changes of an object in multiple video data 200. For example, the data generation unit 118 converts, for each of the multiple video data 200, information on the three-dimensional position where an object exists for each frame into feature vector information.

[0085] The data generation unit 118 generates still image data 300 that uses edges to represent, for example, information on time-series shape changes of an object in multiple video data 200. For example, the data generation unit 118 converts, for each of the multiple video data 200, information on the three-dimensional position where an object exists for each frame into edge information.

[0086] For example, the data generation unit 118 generates text data 400 from the video data 200. The data generation unit 118 may convert information about the time-series movement of an object in the video data 200 into text elements, and generate text data 400 including the converted text elements. The data generation unit 118 may generate text data 400 that represents the information about the time-series movement of an object in the video data 200 using at least one of characters, words, fonts, and ASCII codes. In this way, examples of text elements include, but are not limited to, characters, words, fonts, and ASCII codes.

[0087] The data generating section 118 may convert information on time-series positional changes of multiple objects in the video data 200 into text elements, and generate text data 400 including the converted text elements.

[0088] The data generation unit 118 generates text data 400 that uses characters to represent, for example, information about changes in the positions of multiple objects over time in the video data 200. For example, the data generation unit 118 converts information about the three-dimensional positions of objects in each frame into character information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple objects included in the frame as three consecutive characters. As a result, for the portion of the still image data 300 corresponding to that frame, the three consecutive characters indicate the X, Y, and Z coordinates of the object. The data generation unit 118 can generate text data 400 corresponding to the video data 200 by similarly assigning characters for other frames of the video data 200. The method for converting the X, Y, and Z coordinate information of each of the multiple objects into characters is not limited to this. The X, Y, and Z coordinate information of each of the multiple objects may be compressed and converted into characters using existing compression technology. The frame divisions may be expressed by arrangement, or one or more characters may be used to indicate the frame divisions. Other methods for indicating the frame divisions may also be used.

[0089] Japanese characters may be used as the characters. This results in the generation of Japanese text data 400 representing information on changes in the time series positions of multiple objects in the video data 200. When multiple objects in the video data 200 are moving in a similar manner, the text data 400 may have similar content, with similar characters being repeated. Furthermore, when multiple objects in the video data 200 are moving rapidly for a short period of time, the text data 400 may use characters in only that portion that is different from the characters used in the other portions. Characters from a country other than Japan may be used as the characters. Furthermore, characters from multiple countries may be mixed as the characters.

[0090] The data generation unit 118 generates text data 400 that uses words to represent, for example, information about changes in the positions of multiple objects over time in the video data 200. The data generation unit 118 converts, for example, information about the three-dimensional positions where objects exist for each frame into word information.

[0091] The data generation unit 118 generates text data 400 that uses ASCII code to represent, for example, information on time-series changes in the positions of multiple objects in the video data 200. The data generation unit 118 converts, for example, information on the three-dimensional positions where objects exist for each frame into ASCII code information.

[0092] The data generation unit 118 generates text data 400 that represents, for example, information on time-series positional changes of multiple objects in the video data 200 using at least one of characters and words and fonts. For example, the data generation unit 118 converts information on the three-dimensional positions where objects exist for each frame into information on at least one of characters and words and fonts.

[0093] The data generating unit 118 may convert information on time-series shape changes of objects in the video data 200 into text elements, and generate text data 400 including the converted text elements.

[0094] The data generation unit 118 generates text data 400 that uses characters to represent, for example, information about time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information about the three-dimensional positions of multiple structural points of an object for each frame into character information.

[0095] As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple structural points of an object included in the frame as three consecutive characters. As a result, for the portion of the text data 400 corresponding to that frame, the three consecutive characters indicate the X, Y, and Z coordinates of the object. The data generation unit 118 may generate text data 400 corresponding to the video data 200 by similarly assigning characters for other frames of the video data 200. The method for converting the information on the X, Y, and Z coordinates of each of the multiple structural points of an object into characters is not limited to this. Existing compression technology may be used to compress the information on the X, Y, and Z coordinates of each of the multiple structural points of an object and convert it into characters. Frame divisions may be represented by arrangement, or one or more characters may be used to indicate frame divisions. Methods other than these may also be used to represent frame divisions.

[0096] Japanese characters may be used as the characters. This results in the generation of Japanese text data 400 that represents information on time-series shape changes of objects in the video data 200. When an object in the video data 200 moves in a similar manner, the text data 400 may have similar content, with similar characters being repeated. Furthermore, when an object in the video data 200 moves rapidly for a short period of time, the text data 400 may use characters that are different from those used in other portions. Characters from countries other than Japan may be used as the characters. Furthermore, characters from multiple countries may be mixed as the characters.

[0097] The data generation unit 118 generates text data 400 that uses words to represent, for example, information about time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information about the three-dimensional position where an object exists for each frame into word information.

[0098] The data generation unit 118 generates text data 400 that uses ASCII code to represent, for example, information on time-series shape changes of an object in the video data 200. The data generation unit 118 converts, for example, information on the three-dimensional position where an object exists for each frame into ASCII code information.

[0099] The data generation unit 118 generates text data 400 that represents, for example, information on time-series shape changes of objects in the video data 200 using at least one of characters and words and a font. For example, the data generation unit 118 converts information on the three-dimensional position where an object exists for each frame into information on at least one of characters and words and a font.

[0100] For example, the data generation unit 118 generates text data 400 from a plurality of pieces of video data 200 in which an object is captured from a plurality of different directions. Here, the differences from generating text data 400 from a single piece of video data 200 will be mainly described. The data generation unit 118 may convert information about the time-series movement of an object in the plurality of pieces of video data 200 into text elements, and generate text data 400 including the converted text elements. The data generation unit 118 may generate text data 400 that represents information about the time-series movement of an object in the plurality of pieces of video data 200 using at least one of characters, words, fonts, and ASCII codes.

[0101] The data generating section 118 may convert information on time-series positional changes of a plurality of objects in a plurality of video data 200 into text elements, and generate text data 400 including the converted text elements.

[0102] The data generation unit 118 generates text data 400 that uses characters to represent, for example, information about changes in the positions of multiple objects over time in multiple video data 200. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information about the three-dimensional positions of objects present into character information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X coordinate, Y coordinate, and Z coordinate of each of multiple objects included in the frame as three consecutive characters. As a result, for a portion of the still image data 300 corresponding to that frame, the three consecutive characters indicate the X coordinate, Y coordinate, and Z coordinate of the object. The data generation unit 118 can generate text data 400 corresponding to the video data 200 by similarly assigning characters to other frames of the video data 200.

[0103] The data generation unit 118 generates text data 400 that expresses, by words, information on time-series positional changes of a plurality of objects in a plurality of video data 200. For example, the data generation unit 118 converts, for each of the plurality of video data 200, information on the three-dimensional position where an object exists for each frame into word information.

[0104] The data generation unit 118 generates text data 400 that expresses, for example, information on time-series positional changes of a plurality of objects in a plurality of video data 200 in ASCII code. For example, the data generation unit 118 converts information on the three-dimensional position where an object exists for each frame of each of the plurality of video data 200 into ASCII code information.

[0105] The data generation unit 118 generates text data 400 that expresses, for example, information on time-series positional changes of a plurality of objects in a plurality of video data 200 using at least one of characters and words and fonts. For example, the data generation unit 118 converts information on the three-dimensional position of an object for each frame of each of the plurality of video data 200 into information on at least one of characters and words and fonts.

[0106] The data generating unit 118 may convert information on time-series shape changes of objects in multiple video data 200 into text elements, and generate text data 400 including the converted text elements.

[0107] The data generation unit 118 generates text data 400 that uses characters to represent, for example, information about time-series shape changes of an object in multiple video data 200. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information about the three-dimensional positions of multiple structural points of the object into character information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X coordinate, Y coordinate, and Z coordinate of each of the multiple structural points of the object included in the frame as three consecutive characters. As a result, for the portion of the text data 400 corresponding to that frame, the three consecutive characters indicate the X coordinate, Y coordinate, and Z coordinate of the object. The data generation unit 118 can generate text data 400 corresponding to the video data 200 by similarly assigning characters to other frames of the video data 200.

[0108] The data generation unit 118 generates text data 400 that expresses, by words, information on time-series shape changes of an object in the plurality of video data 200. For example, the data generation unit 118 converts, for each of the plurality of video data 200, information on the three-dimensional position where an object exists for each frame into word information.

[0109] The data generation unit 118 generates text data 400 that expresses, for example, information on time-series shape changes of an object in multiple video data 200 using ASCII code. For example, the data generation unit 118 converts information on the three-dimensional position where an object exists for each frame of each of the multiple video data 200 into ASCII code information.

[0110] The data generation unit 118 generates text data 400 that represents, for example, information on time-series shape changes of objects in a plurality of video data 200 using at least one of characters and words and a font. For example, the data generation unit 118 converts, for each of the plurality of video data 200, information on the three-dimensional position where an object exists for each frame into information on at least one of characters and words and a font.

[0111] For example, the data generation unit 118 generates genome data 500 from the video data 200. The data generation unit 118 may convert information on the time-series movement of an object in the video data 200 into genome elements, and generate genome data 500 including the converted genome elements. The data generation unit 118 may generate genome data 500 that represents information on the time-series movement of an object in the video data 200 using a base sequence.

[0112] The data generation unit 118 may convert information on time-series positional changes of multiple objects in the video data 200 into genome elements, and generate genome data 500 including the converted genome elements.

[0113] The data generation unit 118 generates genome data 500 that represents, for example, information on time-series positional changes of multiple objects in the video data 200 using base sequences. For example, the data generation unit 118 converts information on the three-dimensional positions of objects in each frame into base sequence information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple objects included in the frame as base sequences. As a result, for the portion of the genome data 500 corresponding to that frame, consecutive base sequences indicate the X, Y, and Z coordinates of the object. The data generation unit 118 can generate genome data 500 corresponding to the video data 200 by similarly assigning base sequences for other frames of the video data 200. The method for converting the X, Y, and Z coordinate information of each of the multiple objects into base sequences is not limited to this. Existing compression technology may be used to compress the X, Y, and Z coordinate information of each of the multiple objects and convert it into a base sequence. The frame boundaries may be expressed by the arrangement or by using a base sequence indicating the frame boundaries, or other methods may be used to represent the frame boundaries.

[0114] The data generation unit 118 may convert information on time-series shape changes of objects in the video data 200 into genome elements, and generate genome data 500 including the converted genome elements.

[0115] The data generation unit 118 generates genome data 500 that represents, for example, information on time-series shape changes of an object in the video data 200 using a base sequence. For example, the data generation unit 118 converts, for each frame, information on the three-dimensional positions of multiple structural points of the object into base sequence information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple structural points of the object included in the frame as a base sequence. As a result, for the portion of the genome data 500 corresponding to that frame, the base sequence indicates the X, Y, and Z coordinates of the object. The data generation unit 118 can generate genome data 500 corresponding to the video data 200 by similarly assigning the base sequences for other frames of the video data 200. The method for converting the X, Y, and Z coordinate information of each of the multiple structural points of the object into a base sequence is not limited to this. Existing compression technology may be used to compress the X, Y, and Z coordinate information of each of the multiple structural points of the object and convert it into a base sequence. The frame boundaries may be expressed by the arrangement or by using a base sequence indicating the frame boundaries, or other methods may be used to represent the frame boundaries.

[0116] For example, the data generation unit 118 generates genome data 500 from a plurality of video data 200 in which an object is captured from a plurality of different directions. Here, the differences from generating genome data 500 from a single video data 200 will be mainly described. The data generation unit 118 may convert information on the time-series movement of the object in the plurality of video data 200 into genome elements, and generate genome data 500 including the converted genome elements. The data generation unit 118 may generate text data 400 that represents information on the time-series movement of the object in the plurality of video data 200 using base sequences.

[0117] The data generation unit 118 may convert information on time-series positional changes of multiple objects in multiple video data 200 into genome elements, and generate genome data 500 including the converted genome elements.

[0118] The data generation unit 118 generates genome data 500 that uses base sequences to represent, for example, information on changes in the positions of multiple objects over time in multiple video data 200. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information on the three-dimensional positions of objects present into base sequence information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X, Y, and Z coordinates of each of the multiple objects included in the frame as base sequences. As a result, for the portion of the still image data 300 corresponding to that frame, the base sequences indicate the X, Y, and Z coordinates of the objects. The data generation unit 118 can generate text data 400 corresponding to the video data 200 by similarly assigning base sequences for other frames of the video data 200. <Genome Data: Multiple Videos: Shape Change>

[0119] The data generation unit 118 may convert information on time-series shape changes of objects in multiple video data 200 into genome elements, and generate genome data 500 including the converted genome elements.

[0120] The data generation unit 118 generates genome data 500 that represents, for example, information on time-series shape changes of an object in multiple video data 200 using base sequences. For example, the data generation unit 118 converts, for each frame of the multiple video data 200, information on the three-dimensional positions of multiple structural points of the object into base sequence information. As a specific example, for one frame of the video data 200, the data generation unit 118 assigns the X coordinate, Y coordinate, and Z coordinate of each of the multiple structural points of the object included in the frame as a base sequence. As a result, for a portion of the genome data 500 corresponding to that frame, three consecutive characters indicate the X coordinate, Y coordinate, and Z coordinate of the object. The data generation unit 118 can generate genome data 500 corresponding to the video data 200 by similarly assigning base sequences for other frames of the video data 200.

[0121] The classification result acquisition unit 120 inputs the converted data generated by the data generation unit 118 to the classification network stored in the model storage unit 114 and acquires the classification result output from the classification network. For example, the classification result acquisition unit 120 inputs still image data 300 generated by the classification result acquisition unit 120 to the still image classification network and acquires the classification result output from the still image classification network. For example, the classification result acquisition unit 120 inputs text data 400 generated by the classification result acquisition unit 120 to the text classification network and acquires the classification result output from the text classification network. For example, the classification result acquisition unit 120 inputs genome data 500 generated by the classification result acquisition unit 120 to the genome classification network and acquires the classification result output from the genome classification network.

[0122] The model generation unit 122 generates a classification network that classifies the converted data by performing machine learning using the converted data generated by the data generation unit 118. The model generation unit 122 may store the generated classification network in the model storage unit 114. For example, the model generation unit 122 generates a still image classification network that classifies still image data by performing machine learning using still image data 300 generated by the data generation unit 118. For example, the model generation unit 122 generates a text classification network that classifies text data by performing machine learning using text data 400 generated by the data generation unit 118. For example, the model generation unit 122 generates a genome classification network that classifies genome data by performing machine learning using genome data 500 generated by the data generation unit 118.

[0123] The output unit 130 executes various output controls and may include an alert output unit 132, a display output unit 134, and an audio output unit 136.

[0124] The alert output unit 132 outputs an alert according to the classification result obtained by the classification result acquisition unit 120. The alert output unit 132 may display and output alert information on a display provided in the data processing device 100, output an alert as sound from a speaker provided in the data processing device 100, display and output alert information on a display provided in the communication terminal 30, or output an alert as sound from a speaker provided in the communication terminal 30.

[0125] For example, the alert output unit 132 outputs an alert when the classification results acquired by the classification result acquisition unit 120 satisfy a predetermined condition. For example, in a situation where the video data acquisition unit 116 continuously acquires video data 200, the data generation unit 118 continuously generates converted data, and the classification result acquisition unit 120 continuously acquires classification results, the alert output unit 132 outputs an alert when the degree of difference between successive classification results exceeds a predetermined threshold. Furthermore, for example, after multiple pieces of converted data generated by the data generation unit 118 are accumulated, the classification result acquisition unit 120 collectively acquires classification results for the multiple converted data, and the alert output unit 132 outputs an alert when the classification results include a classification result that indicates an abnormal value.

[0126] The display output unit 134 may display and output the converted data generated by the data generation unit 118. For example, the display output unit 134 displays and outputs the converted data generated by the data generation unit 118 on a display included in the data processing device 100. For example, the display output unit 134 displays and outputs the converted data generated by the data generation unit 118 on a display included in the communication terminal 30.

[0127] For example, the display output unit 134 displays and outputs still image data 300 generated by the data generation unit 118. The display output unit 134 may continuously display and output the still image data 300 continuously generated by the data generation unit 118. For example, the display output unit 134 displays and outputs text data 400 generated by the data generation unit 118. The display output unit 134 may continuously display and output the text data 400 continuously generated by the data generation unit 118. For example, the display output unit 134 displays and outputs genome data 500 generated by the data generation unit 118. The display output unit 134 may continuously display and output the genome data 500 continuously generated by the data generation unit 118. These features can make a viewer viewing the display aware that some abnormality may have occurred in the object.

[0128] The display output unit 134 may display and output the classification results acquired by the classification result acquisition unit 120. For example, the display output unit 134 displays and outputs the classification results acquired by the classification result acquisition unit 120 on a display included in the data processing device 100. For example, the display output unit 134 displays and outputs the classification results acquired by the classification result acquisition unit 120 on a display included in the communication terminal 30.

[0129] The audio output unit 136 may output the converted data generated by the data generation unit 118 as audio. The display output unit 134 may output, for example, the text data 400 generated by the data generation unit 118 as audio. The audio output unit 136 may output the text data 400 generated by the data generation unit 118 as audio from a speaker included in the data processing device 100. The audio output unit 136 may output the text data 400 generated by the data generation unit 118 as audio from a speaker included in the communication terminal 30. This makes it possible for a person who hears the audio to become aware that some kind of abnormality may have occurred in the object.

[0130] The converted data acquisition unit 142 acquires converted data including elements converted from information on the time-series movement of an object in the video data. The converted data acquisition unit 142, for example, acquires still image data including still image elements converted from information on the time-series movement of an object in the video data. The converted data acquisition unit 142, for example, acquires text data including text elements converted from information on the time-series movement of an object in the video data. The converted data acquisition unit 142, for example, acquires genome data including genome elements converted from information on the time-series movement of an object in the video data.

[0131] The video data generation unit 144 generates video data from the converted data acquired by the converted data acquisition unit 142. When the converted data acquisition unit 142 acquires the converted data generated by the data generation unit 118, the video data generation unit 144 generates video data from the converted data using the conversion information used to generate the converted data, which is stored in the conversion information storage unit 112. The video data generation unit 144 generates video data 200 from, for example, still image data 300 generated by the data generation unit 118. The video data generation unit 144 generates video data 200 from, for example, text data 400 generated by the data generation unit 118. The video data generation unit 144 generates video data 200 from, for example, genome data 500 generated by the data generation unit 118.

[0132] When converted data generated by another device is acquired, the converted data acquisition unit 142 may also acquire conversion information used by the other device when generating the converted data. The video data generation unit 144 may generate video data from the converted data using the conversion information. The video data generation unit 144 generates video data from, for example, still image data generated by the other device. The video data generation unit 144 generates video data from, for example, text data generated by the other device. The video data generation unit 144 generates video data from, for example, genome data generated by the other device.

[0133] The display output unit 134 may display and output the video data generated by the video data generation unit 144. For example, the display output unit 134 displays and outputs the video data generated by the video data generation unit 144 on a display included in the data processing device 100. For example, the display output unit 134 displays and outputs the video data generated by the video data generation unit 144 on a display included in the communication terminal 30.

[0134] It should be noted that the data processing device 100 does not necessarily have to include all of the conversion information storage unit 112, the model storage unit 114, the video data acquisition unit 116, the data generation unit 118, the classification result acquisition unit 120, the model generation unit 122, the output unit 130, the conversion data acquisition unit 142, and the video data generation unit 144. As described above, the data processing device 100 may have the function of generating conversion data from video data but not the function of generating video data from the conversion data. In this case, the data processing device 100 does not have to include the conversion data acquisition unit 142 and the video data generation unit 144. Furthermore, the data processing device 100 may have the function of generating video data from conversion data but not the function of generating conversion data from video data. In this case, the data processing device 100 does not have to include the model storage unit 114, the video data acquisition unit 116, the data generation unit 118, the classification result acquisition unit 120, and the model generation unit 122.

[0135] 9 schematically illustrates an example of the hardware configuration of a computer 1200 functioning as the data processing device 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or can cause the computer 1200 to perform operations associated with the device according to the present embodiment or one or more "parts" thereof, and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0136] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0137] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0138] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0139] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0140] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0141] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0142] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0143] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0144] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0145] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0146] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0147] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0148] Computer-readable instructions may be provided locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0149] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0150] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0151] 20 Network, 30 Communication terminal, 32 Camera, 40 Aquaculture farm, 100 Data processing device, 102 Camera, 112 Conversion information storage unit, 114 Model storage unit, 116 Video data acquisition unit, 118 Data generation unit, 120 Classification result acquisition unit, 122 Model generation unit, 130 Output unit, 132 Alert output unit, 134 Display output unit, 136 Audio output unit, 142 Conversion data acquisition unit, 144 Video data generation unit, 200 Video data, 210, 220, 230, 240, 250, 260 Video data, 211, 221, 231, 241, 251, 261 Frame, 300 Still image data, 302, 311, 321, 331, 341, 351, 361 Part, 400 Text data, 402, 411, 421, 431, 441, 451, 461 parts, 500 genome data, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input / output chip

Claims

1. A data processing device comprising: a video data acquisition unit that acquires video data of an object; and a data generation unit that converts information on the time-series movement of the object in the video data into still image elements, and generates still image data including the elements of the converted still image.

2. A data processing device according to claim 1, wherein the data generation unit generates the still image data that represents information on the time-series movement of the object in the video data by at least one of pixel values, feature vectors, and edges.

3. A data processing device as described in claim 1, wherein the data generation unit converts information on changes in the time series positions of the objects in the video data into the elements of the still image, and generates the still image data including the elements of the converted still image.

4. A data processing device as described in claim 1, wherein the data generation unit converts information on time-series shape changes of the object in the video data into the elements of the still image, and generates the still image data including the elements of the converted still image.

5. A data processing device according to claim 1, wherein the data generation unit converts information on the time-series movement of the object in the video data into text elements, and generates text data including the converted text elements.

6. A data processing device as described in claim 1, wherein the data generation unit converts information on the time-series movement of the object in the video data into genome elements, and generates genome data including the converted genome elements.

7. A data processing device as described in any one of claims 1 to 6, further comprising: a classification result acquisition unit that inputs the still image data generated by the data generation unit into a still image classification network trained to classify still image data, and acquires the classification result output from the still image classification network; and an alert output unit that outputs an alert when the classification result acquired by the classification result acquisition unit satisfies a predetermined condition.

8. A data processing device according to any one of claims 1 to 6, further comprising: a conversion data acquisition unit that acquires still image data including still image elements converted from information on the time-series movement of an object in video data; and a video data generation unit that generates the video data from the still image data.

9. A data processing device comprising: a still image data acquisition unit that acquires still image data including still image elements converted from information on the time-series movement of an object in video data; and a video data generation unit that generates the video data from the still image data.

10. A data processing device comprising: a video data acquisition unit that acquires video data of an object; and a data generation unit that converts information on the time-series movement of the object in the video data into text elements, and generates text data including the converted text elements.

11. A data processing device comprising: a video data acquisition unit that acquires video data of an object; and a data generation unit that converts information on the time-series movement of the object in the video data into genome elements, and generates genome data including the converted genome elements.

12. A program for causing a computer to function as a data processing device according to any one of claims 1 to 6, 9, 10 and 11.

Images