Video processing device, video processing system, video processing method and program

Abstract

The present disclosure provides a video processing device, a video processing system, and a video processing method that are capable of suppressing the influence of changes occurring in image quality of a video and improving the accuracy of video recognition. A video processing device (10) according to one aspect of the present disclosure comprises a characteristic information generation unit (11) that generates image quality characteristic information indicating spatio-temporal characteristics of image quality information indicating an image quality of a video, an integration unit (12) that produces integrated data by integrating video information including spatio-temporal characteristics of the video with the image quality characteristic information generated by the characteristic information generation unit (11), and a recognition unit (13) that executes recognition processing on a subject shown in the video on the basis of the integrated data.

Description

[Technical Field]

[0001] The present disclosure relates to a video processing device, a video processing system, and a video processing method. [Background technology]

[0002] Video processing technology has been advancing in recent years.

[0003] For example, Patent Document 1 discloses a method for identifying a predetermined object from image data that may include the object in a cloud server. Specifically, when encoding video data including image data, the cloud server generates encoding parameter features, which are features regarding mapping information in which encoding parameters determined for each unit image area are mapped to the unit image area, and image features, which are features related to pixel values ​​of the image data. The cloud server then inputs the generated encoding parameter features and image features into a trained classifier, which then outputs information related to the class of the predetermined object, thereby identifying the object from the image data.

[0004] Patent Document 2 also discloses a moving image processing device, which performs quantization processing of the face region such that when the area ratio of the face region to the entire input image is relatively large, the reduction in the compression rate in the face region is small, and when the area ratio of the face region to the entire input image is relatively small, the reduction in the compression rate in the face region is large. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-043773 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-193441 Summary of the Invention [Problem to be solved by the invention]

[0006] If the image quality of the image used in the recognition process changes over time, there is a possibility that the recognition engine will not be able to accurately recognize the changed image. The technology disclosed in Patent Document 1 aims to reduce the processing load by using "encoding parameter features" in the recognition process, but it does not solve this problem. Furthermore, the technology disclosed in Patent Document 2, which balances the compression rate between the face area and other areas, does not solve this problem either.

[0007] An object of the present disclosure is to provide an image processing device, an image processing system, and an image processing method that can suppress the impact of changes in image quality in an image and improve the accuracy of image recognition. [Means for solving the problem]

[0008] An image processing device according to one aspect of this embodiment includes a feature information generation unit that generates image quality feature information that indicates the spatiotemporal features of image quality information that indicates the image quality of an image, an integration unit that generates integrated data that integrates information about the image that includes the spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit, and a recognition unit that performs recognition processing on a subject included in the image based on the integrated data.

[0009] An image processing system according to one aspect of this embodiment includes a feature information generation unit that generates image quality feature information that indicates the spatiotemporal features of image quality information that indicates the image quality of an image, an integration unit that generates integrated data that integrates information about the image that includes the spatiotemporal features of the image and the image quality feature information generated by the feature information generation unit, and a recognition unit that performs recognition processing on a subject included in the image based on the integrated data.

[0010] One aspect of the image processing method according to this embodiment is executed by a computer, and generates image quality feature information indicating the spatiotemporal characteristics of image quality information that indicates the image quality of the image, generates integrated data that integrates information about the image that includes the spatiotemporal characteristics of the image with the image quality feature information, and performs recognition processing on the subject included in the image based on the integrated data. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide an image processing device, an image processing system, and an image processing method that can suppress the impact of changes in image quality and improve the accuracy of image recognition even when changes occur in the image quality. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a video processing device according to a first embodiment; [Figure 2] 4 is a flowchart illustrating an example of a typical process of the video processing device according to the first embodiment. [Figure 3] 1 is a block diagram illustrating an example of a video processing system according to a first embodiment. [Figure 4] FIG. 10 is a block diagram illustrating an example of a video recognition system according to a second embodiment. [Figure 5A] FIG. 10 is a block diagram showing an example of a center server according to a second embodiment. [Figure 5B] FIG. 11 is a block diagram illustrating an example of a compressed information integration unit according to a second embodiment. [Figure 6A] FIG. 10 is a diagram illustrating an example of QP map information. [Figure 6B] FIG. 10 is a diagram illustrating an example of generated attention map information. [Figure 7] 10 is a flowchart showing an example of a representative process of a center server according to the second embodiment. [Figure 8] FIG. 11 is a block diagram illustrating another example of the compressed information integration unit according to the second embodiment. [Figure 9]FIG. 2 is a block diagram illustrating an example of a hardware configuration of an apparatus according to each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, each embodiment will be described with reference to the drawings. Note that the following description and drawings have been omitted or simplified as appropriate for clarity of explanation.

[0014] Embodiment 1 (1A) Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. In this (1A), a video processing device will be described.

[0015] 1 is a block diagram showing an example of a video processing device. The video processing device 10 includes a feature information generating unit 11, an integrating unit 12, and a recognizing unit 13. Each unit (means) of the video processing device 10 is controlled by a control unit (controller) not shown. Each unit will be described below.

[0016] [Configuration Description] The feature information generation unit 11 generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video. The video is data on which recognition processing regarding a subject is performed, and is assumed to be, for example, data acquired by a camera or the like, but is not limited to this. The video is data that includes multiple still images (hereinafter simply referred to as images) in a time series. Note that in the present disclosure, the terms video and image are interchangeable. That is, the video processing device 10 can be referred to as a video processing device that processes video, or as an image processing device that processes images. The video processing device 10 can acquire this video, for example, from outside the video processing device.

[0017] The image quality information is any information indicating image quality, and may be, for example, information indicating the degree of compression of a region of a frame (image frame) included in a video, or image brightness information or luminance information, etc. An example of the information indicating the degree of compression of a region of a frame included in a video is, but is not limited to, a QP (Quantization Parameter) map, which is a map of spatiotemporal features of image quality information.

[0018] The integration unit 12 generates integrated data by integrating information about the video, including the spatiotemporal features of the video, with the image quality feature information generated by the feature information generation unit 11. The information about the video may be information indicating the spatiotemporal features of the video after any processing has been performed on the video (video feature information), or it may be the video itself. More specifically, the video feature information is a feature amount related to pixel values ​​of the video, and can be expressed, for example, by a matrix indicating the feature amount. The video feature information may be generated by the video processing device 10 based on the video, or may be generated by a device external to the video processing device 10.

[0019] Furthermore, the integrating unit 12 can use any method for integration as long as the integrated data reflects image quality feature information on information related to the video. For example, the integration may be performed by any arithmetic process such as multiplication or addition, may be performed by an algorithm based on a predefined rule base, or may be performed by a pre-trained AI (Artificial Intelligence) model such as a neural network. Details of this will be described later in the second embodiment.

[0020] The recognition unit 13 performs recognition processing on the subject included in the video based on the integrated data generated by the integration unit 12. The recognition unit 13 is capable of performing any recognition processing on the subject, and may, for example, identify the attributes of the subject. The attributes of the subject may indicate the type of object defined for the subject, such as whether the subject is a human, a non-human organism, or a machine such as a bicycle, automobile, or robot. Furthermore, if the subject is a human, the attributes of the subject may be information that uniquely identifies the subject, such as whether the subject is one of persons A, B, C, etc., pre-stored in the video processing device 10, or whether the subject is an unknown person who is not stored. Furthermore, if the subject is a human, the attributes of the subject may be information that identifies the occupation of the subject (for example, whether the subject is a construction worker, a plasterer, or an ordinary passerby). If the subject is a machine, the attributes of the subject may be information that identifies the type of machine, such as whether the subject is a bicycle, automobile, or industrial robot. As another example, the recognition unit 13 may identify the movement the subject is performing. The movement of the subject is, for example, the behavior of the subject when the recognition unit 13 identifies the subject as a human, and the work content of the subject when the recognition unit 13 identifies the subject as a robot.

[0021] Note that the recognition unit 13 may be, for example, an AI model (e.g., a neural network) that has been trained in advance. The training is performed by inputting sample videos including a subject and training data including a correct answer label indicating what the subject is or a correct answer label indicating the movement of the subject for each video into the recognition unit 13 (or the video processing device 10). Alternatively, the recognition unit 13 may analyze the video based on a predefined rule base to determine what the subject is or the movement of the subject.

[0022] [Processing Description] 2 is a flowchart showing an example of a typical process of the video processing device 10, and this flowchart will be used to explain an overview of the process of the video processing device 10. Note that the details of each process are as described above, and therefore will not be explained again.

[0023] First, the feature information generation unit 11 generates image quality feature information that indicates the spatiotemporal features of image quality information that indicates the image quality of the video (step S11; generation step). The integration unit 12 generates integrated data by integrating information about the video and the image quality feature information generated by the feature information generation unit (step S12; integration step). The recognition unit 13 executes recognition processing for the subject included in the video based on the integrated data (step S13; recognition step).

[0024] [Effect description] As described above, the recognition unit 13 can perform recognition processing on a subject based on integrated data on the video that reflects image quality feature information. In other words, even if a change in image quality occurs in the video, the recognition unit 13 can perform recognition processing after grasping the information as image quality feature information. Therefore, the influence of the change in image quality that occurs in the video can be suppressed, and the accuracy of video recognition can be improved.

[0025] (1B) Next, in (1B), a video processing system will be described. FIG. 3 is a block diagram showing an example of a video processing system. Video processing system 20 includes a feature information generation device 21 and a recognition device 22. Feature information generation device 21 has a feature information generation unit 11, and recognition device 22 has an integration unit 12 and a recognition unit 13. These feature information generation units 11 to 13 perform the same processing as shown in (1A). When feature information generation unit 11 generates image quality feature information, the generated image quality feature information is output to recognition device 22. The integration unit 12 uses the image quality feature information to perform the processing shown in (1A).

[0026] As described above, the video processing according to the present disclosure may be implemented by a single device, as shown in (1A), or may be implemented as a system in which the processing is distributed across multiple devices, as shown in (1B). Note that the device configuration shown in (1B) is merely exemplary. As another example, a first device may have a feature information generator 11 and an integration unit 12, and a second device may have a recognition unit 13. Alternatively, three different devices may be provided, each having a feature information generator 11, an integration unit 12, and a recognition unit 13. As yet another example, the video processing system 20 may be partially or entirely provided on a cloud server built on a cloud, or on another type of virtualized server created using virtualization technology, etc. Functions other than those provided on such servers are located at the edge. For example, in a system for monitoring video captured on-site via a network, the edge is a device located at or near the site, and is a device close to the terminal in the network hierarchy.

[0027] Embodiment 2 In the following embodiment 2, a specific example of the video processing device 10 described in embodiment 1 will be disclosed. However, the specific example of the video processing device 10 described in embodiment 1 is not limited to the one shown below. Furthermore, the configurations and processes described below are examples and are not limited to these.

[0028] (2A) [Configuration Description] Figure 4 is a block diagram showing an example of a video recognition system. Video recognition system 100 includes terminal 101, base station 102, MEC (Multi-access Edge Computing) server 103, and center server 104. In the example of Figure 4, terminal 101 is provided on the edge side (field side) of video recognition system 100, and center server 104 is located at a location away from the field (cloud side). Each device will be described below.

[0029] Terminals 101A, 101B, and 101C (hereinafter collectively referred to as terminals 101) are edge devices connected to a network, have a camera as a photographing unit, and can photograph any location. Terminal 101 transmits the photographed video to center server 104 via base station 102. In this example, terminal 101 transmits the video via a wireless line. However, the video may also be transmitted via a wired line.

[0030] However, the terminal 101 and the camera may be provided separately. In this case, the camera transmits the captured video to the terminal 101, which is a relay device, and the terminal 101 processes the video as necessary and transmits it to the center server 104 via the base station 102. However, the camera may process the video and transmit it to the terminal 101, and the terminal 101 may then transmit the video.

[0031] Furthermore, as will be described later, each terminal 101 is assigned a bit rate for video that can be transmitted to the center server 104 by the MEC server 103. The video bit rate means the amount of video data per unit time (e.g., 1 second). The assigned bit rate can change over time. Each terminal 101 can reduce (i.e., compress) the bit rate of a part or all of the captured video by a predetermined percentage so that the bit rate of the video transmitted to the center server 104 is equal to or less than the assigned bit rate.

[0032] Furthermore, when the terminal 101 detects that a predetermined condition is satisfied, it can reduce the bit rate of a partial region or the entire region of a frame of a captured video by a predetermined percentage. The terminal 101 may perform this process by, for example, analyzing the captured video. In particular, when the terminal 101 detects that a predetermined object (e.g., a predetermined person) is included in a frame of a captured video, it can reduce the bit rate of regions other than that region by a predetermined percentage compared to the bit rate of that region. However, the terminal 101 can also reduce the bit rate of regions including the predetermined object by a predetermined percentage compared to the bit rate of regions other than that region. As another example, when the terminal 101 detects that it is in a predetermined environment (e.g., the video is captured during a predetermined time period), it can reduce the bit rate of the entire frame of the captured video by a predetermined percentage.

[0033] In this way, when the terminal 101 executes video compression under predetermined conditions, the terminal 101 generates QP map information, which is information indicating the degree of compression of frame regions included in the video, and transmits the QP map information to the base station 102. The terminal 101 may also uniformly compress the video to be transmitted so that it can be decompressed later by the center server 104.

[0034] The base station 102 transfers video transmitted from each terminal 101 via the network to the center server 104. The base station 102 also transfers control signals from the MEC server 103 to each terminal 101. For example, the base station 102 is a local 5G (5th Generation) base station, a 5G gNB (next generation Node B), an LTE eNB (evolved Node B), a wireless LAN access point, or the like, but may also be another relay device. The network is, for example, a core network such as 5GC (5th Generation Core network) or EPC (Evolved Packet Core), the Internet, or the like.

[0035] The MEC server 103 allocates a bit rate for the video that each terminal 101 transmits to the base station 102, and transmits information about the allocated video bit rate to each terminal 101 as control information. Each terminal 101 adjusts the video bit rate according to the control information, as described above. Note that the base station 102 and the MEC server 103 are connected to each other so that they can communicate with each other using any communication method, but the base station 102 and the MEC server 103 may also constitute a single device.

[0036] The MEC server 103 detects at least one of the communication environment between each terminal 101 and the base station 102 or the communication environment between the base station 102 and the MEC server 103, and based on the detection results, determines the video bit rate to be allocated to each terminal 101. At this time, the MEC server 103 predicts the accuracy with which the center server 104 (described below) will recognize an object based on the video captured by each terminal 101, and can determine the video bit rate to be allocated to each terminal 101 so that the predicted recognition accuracy for the video captured by each terminal 101 is maximized in total.

[0037] The MEC server 103 transmits information about the determined bit rate as control information to each terminal 101. Each terminal 101 adjusts the bit rate of the video to be transmitted to the center server 104 based on this control information.

[0038] The communication environment between each terminal 101 and base station 102 may be determined by, for example, at least one of the number of terminals 101, the degree of congestion in wireless communication between each terminal 101 and base station 102, or the quality of wireless communication. An example of the degree of congestion in wireless communication is the number of packets per unit time, and an example of the quality of wireless communication is radio wave strength (RSSI: Received Signal Strength Indicator), but these are not limited to these. The communication environment between the base station 102 and MEC server 103 may be determined by, for example, at least one of the degree of congestion in wireless communication between the base station 102 and MEC server 103 or the quality of wireless communication. The MEC server 103 can detect at least one of the communication environment between each terminal 101 and base station 102 or the communication environment between the base station 102 and MEC server 103 using one or more parameters described above.

[0039] Furthermore, the MEC server 103 may set predetermined conditions for reducing the bit rate of a part or all of the area of ​​the video captured by the terminal 101, and transmit the setting information to each terminal 101. When the terminal 101 detects that the predetermined conditions have been met based on the setting information, it can reduce the bit rate of a part or all of the area of ​​the captured video.

[0040] As described above, the video recognition system 100 can reduce the bit rate of video transmitted from the terminal 101 in certain cases. This makes it possible to reduce the processing load when processing is performed on the center server 104 side and the communication load within the system. However, due to fluctuations in network communication quality, there is a possibility that video from the terminal 101 will not be transmitted with high quality or accurately. Furthermore, when video, which is time-series data, is transmitted from the terminal 101, block noise may occur due to fluctuations in communication quality, etc. For these reasons, if changes occur in the image quality of the video, there is a possibility that the recognition accuracy of the video will decrease when analyzing the video. However, the center server 104 described below can suppress such phenomena.

[0041] 5A is a block diagram showing an example of a center server. The center server 104 includes a video acquisition unit 111, a QP map information acquisition unit 112, a compressed information integration unit 113, and a behavior recognition unit 114. The center server 104 executes the following video processing for each terminal 101. Each unit of the center server 104 will be described below.

[0042] Video acquisition unit 111 is an interface that acquires video transmitted from each terminal 101 via base station 102 and QP map information corresponding to the video. As described in the first embodiment, QP map information is information that indicates the degree of compression of a frame area included in the video. Note that if the video transmitted from each terminal 101 is uniformly compressed, video acquisition unit 111 executes a decompression process to enable a recognition process, which will be described later, to be executed. Video acquisition unit 111 outputs the acquired information to QP map information acquisition unit 112 and compression information integration unit 113.

[0043] QP map information acquisition unit 112 extracts and acquires QP map information indicating the degree of compression of the video bit rate from the information acquired from video acquisition unit 111. Note that if QP map information is not transmitted from terminal 101, QP map information acquisition unit 112 can acquire QP map information corresponding to the video by analyzing the video output from video acquisition unit 111. QP map information acquisition unit 112 outputs the acquired QP map information to compression information integration unit 113.

[0044] The compressed information integration unit 113 generates integrated data for each frame of the video by integrating the video and the image quality feature information created based on the QP map information, and outputs the data to the behavior recognition unit 114. Details of this will be described later.

[0045] The behavior recognition unit 114 corresponds to the recognition unit 13 according to the first embodiment, and recognizes the behavior of a person who is a subject of the video by analyzing the integrated data output from the compressed information integration unit 113. The behavior recognition unit 114 may be an AI model (e.g., a neural network) that has been trained in advance. The method of this training is the same as that of the recognition unit 13, and therefore a description thereof will be omitted. Alternatively, the behavior recognition unit 114 may determine the movement of the subject by analyzing the video based on a predefined rule base.

[0046] 5B is a block diagram showing an example of the compressed information integration unit 113. The compressed information integration unit 113 has a feature information generation unit 120 having an attention map generation unit 121, and a feature integration unit 122. Each unit of the compressed information integration unit 113 will be described below.

[0047] The feature information generation unit 120 corresponds to the feature information generation unit 11 according to the first embodiment. The attention map generation unit 121 included in the feature information generation unit 120 uses the QP map information output from the QP map information acquisition unit 112 to generate, for each frame of a video, attention map information indicating an area within the frame that should be focused on in recognition processing (hereinafter also referred to as an attention area). The attention map information is a map of spatiotemporal feature amounts of the QP map information. An example of how the attention map generation unit 121 generates attention map information will be described below with reference to FIGS. 6A and 6B.

[0048] 6A is a diagram showing an example of QP map information, showing QP map information (QP map sequence) for each frame in a time series of times T=t1, t2, t3, .... In the QP map at each time, F1 to F3 indicate the area of ​​the entire frame. Therefore, the QP map information indicates spatiotemporal information.

[0049] In FIG. 6A, hatched areas H1 and H2 in frame F2 are areas that are more highly compressed than the other areas in frame F2. For example, it is assumed that terminal 101 performs processing to reduce the video bit rate for hatched areas H1 and H2, but does not perform processing to reduce the video bit rate for the other areas. Alternatively, terminal 101 may perform processing to significantly reduce the video bit rate for hatched areas H1 and H2, but perform processing to reduce the degree of bit rate reduction for the other areas compared to hatched areas H1 and H2. Similarly, hatched area H3 in frame F3 is also an area that is more highly compressed than the other areas in frame F3. In this way, the QP map sequence indicates the degree of compression of the video bit rate in time and space.

[0050] In the QP map sequence, the positions and sizes of regions with high and low compression levels change over time. For example, at one time, there may be regions with high compression levels throughout the entire frame, at another time, there may be regions with low compression levels throughout the entire frame, and at yet another time, there may be a mixture of regions with high and low compression levels within the frame.

[0051] Since the bit rate of the video is reduced in the hatched areas H1 to H3, it is considered difficult to perform accurate recognition processing (inference processing) for those areas even if the video of those areas is input to the behavior recognition unit 114. Furthermore, including such areas in the recognition processing would increase the processing load on the center server 104.

[0052] The attention map generation unit 121 determines whether or not there is a region in the QP map for each time shown in FIG. 6A where the bit rate has decreased from the reference value by a predetermined threshold or more. If there is a region where the degree of decrease in bit rate is equal to or greater than the predetermined threshold, the attention map generation unit 121 excludes that region from the region of interest. In other words, the attention map generation unit 121 masks that region. On the other hand, if there is a region where the degree of decrease in bit rate is less than the predetermined threshold, the attention map generation unit 121 leaves that region as a region of interest (i.e., a region valid for inference processing). Note that information on the reference value and threshold used for the determination is stored in a storage unit (not shown) in the center server 104, and the attention map generation unit 121 acquires that information when performing this determination.

[0053] FIG. 6B is a diagram showing an example of attention map information generated by the attention map generation unit 121 based on the QP map information shown in FIG. 6A, and shows attention map information (attention map sequence) for each frame in the time series of time T=t1, t2, t3, .... In the QP map at each time, F1 to F3 indicate the area of ​​the entire frame. In this case, hatched areas H1 to H3 have been determined as areas where the degree of bit rate reduction is equal to or greater than a predetermined threshold, and therefore have been excluded from the area in the attention map sequence. In this example, weighting is performed so that the weight of each pixel information in the excluded area in the attention map sequence is "0" and the weight of each pixel information in each pixel in the other areas is "1."

[0054] Note that pixel information refers to values ​​stored for a specific unit area in a frame of an image or attention map. For example, pixel values ​​(such as actual RGB values ​​stored in each pixel of an image) may be used, but are not limited to these. The attention map generation unit 121 uses the QP map sequence to define weighting as described above for each unit area in each frame of the time series, so that the weight is either "0" or "1." For example, the attention map generation unit 121 may set the hatched area H1 as one unit area and define the weight of that area as "0." Alternatively, the attention map generation unit 121 may set the unit areas so that the hatched area H1 is composed of multiple unit areas and define the weight of each unit area as "0." In this case, the unit area is composed of one or multiple pixels. The attention map generation unit 121 outputs this attention map information to the feature integration unit 122.

[0055] The feature integration unit 122 corresponds to the integration unit 12 according to the first embodiment and integrates the generated attention map information with the video. The feature integration unit 122 may generate the integrated data by, for example, multiplying each pixel information of the attention map information at each time by each pixel information (e.g., pixel value information) of the corresponding video. In the example of attention map information described above, the weight of each pixel information in the excluded region is "0," so the information for each pixel in this region is also "0" in the integrated data. Therefore, the integrated data includes an image in which the excluded region is masked, and this image represents the region that should be focused on for recognition processing.

[0056] The feature integration unit 122 outputs the integrated data in which the attention areas have been weighted in time and space to the behavior recognition unit 114. The behavior recognition unit 114 performs recognition processing based on this integrated data. In this recognition processing, areas other than the attention areas are prevented from being targeted for the recognition processing, and areas of high-quality video that are easy to analyze are targeted for the recognition processing. This not only increases the accuracy of the recognition processing, but also reduces the processing load of the recognition processing.

[0057] [Processing Description] 7 is a flowchart showing an example of a typical process of the center server 104, and this flowchart will be used to explain an overview of the process of the center server 104. Note that the details of each process are as described above, and therefore will not be explained again.

[0058] First, the video acquisition unit 111 acquires video transmitted from each terminal 101 and QP map information corresponding to the video (step S21; acquisition step). The QP map information acquisition unit 112 extracts QP map information from the information acquired from the video acquisition unit 111 (step S22; extraction step).

[0059] The attention map generation unit 121 generates attention map information using the extracted QP map (step S23; generation step). The feature integration unit 122 integrates the generated attention map information with the video to generate integrated data (step S24; integration step). The behavior recognition unit 114 executes recognition processing based on this integrated data (step S25; recognition step).

[0060] [Effect description] As described above, the attention map generation unit 121 generates attention map information (image quality feature information) indicating spatiotemporal features using QP map information (image quality information) indicating the image quality of the video. The feature integration unit 122 generates integrated data by integrating the video and the attention map information, and the behavior recognition unit 114 performs recognition processing on the subject included in the video based on the integrated data. The behavior recognition unit 114 can perform recognition processing after identifying areas in the video where the bit rate drops significantly. This makes it possible to suppress the impact of changes in image quality that occur in the video and improve the accuracy of video recognition.

[0061] Furthermore, the attention map generation unit 121 may generate attention map information indicating weights of pixel information in a video frame based on the QP map information. The feature integration unit 122 generates integrated data of a video in which the pixels of the video frame have been weighted based on the attention map information. This allows the behavior recognition unit 114 to analyze the integrated data using a method similar to that used for normal video, eliminating the need for a special behavior recognition function to be installed in the center server 104 and reducing costs.

[0062] Furthermore, QP map information, which indicates the degree of compression of a frame region included in a video, may be used as image quality information indicating the image quality of the video. This prevents the behavior recognition unit 114 from analyzing regions with a high degree of compression. Therefore, as described above, it is possible to improve the accuracy of the recognition process and reduce the processing load of the recognition process.

[0063] The behavior recognition unit 114 may recognize the behavior of the subject. For the reasons described above, the behavior recognition unit 114 can determine the behavior of the subject with high accuracy.

[0064] In (2A), the attention map generation unit 121 can generate attention map information from QP map information by determining an algorithm based on a rule base using a threshold, as described above. However, the attention map generation unit 121 may also be a pre-trained AI model (e.g., a neural network). This training is performed by inputting training data including sample QP map information and correct labels indicating attention map information corresponding to each frame of the sample QP map information into the AI ​​model. This method also allows the attention map generation unit 121 to generate attention map information in which areas where accurate recognition processing is considered difficult are masked.

[0065] Below, (2B) and (2C) describe variations of (2A).

[0066] (2B) In (2A), the attention map generation unit 121 generates attention map information by masking areas where the degree of decrease in bit rate from the reference value is equal to or greater than a predetermined threshold. However, even such areas may be useful for behavior recognition processing in some cases. Therefore, in (2B), a variation for generating attention map information that takes such areas into consideration will be described.

[0067] In detail, in (2A), the attention map generating unit 121 sets the weight of each pixel information of an area where the degree of bit rate reduction is equal to or greater than a predetermined threshold to "0" to mask the area. However, the attention map generating unit 121 does not necessarily set the weight of the pixel information of the area where the degree of bit rate reduction is equal to or greater than a predetermined threshold to "0", but may set it to a value greater than 0 and less than 1. In this case, the weight of the information of the area where the degree of bit rate reduction is equal to or greater than a predetermined threshold is low, but the area will be subject to recognition processing by the behavior recognition unit 114.

[0068] In this example, the attention map generation unit 121 is a neural network that has been trained in advance. When training this neural network, a sample video including a plurality of sample images is input to the center server 104 as the video. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 perform the above-described processing on the acquired sample video. At this time, the attention map generation unit 121 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct label of the behavior recognition corresponding to the sample video is equal to or less than a predetermined threshold. For example, the loss function may be trained to have the smallest possible value. The loss function may be, for example, a cross-entropy loss or a mean squared error, but is not limited to these. Through this training, the weighting settings in the attention map generation unit 121 are updated so that the weight of pixel information is a value other than "0" depending on the situation, even in areas where the degree of bit rate reduction is equal to or greater than a predetermined threshold.

[0069] The feature integration unit 122 integrates the attention map information generated by the attention map generation unit 121 as described above with the video. As described above, the feature integration unit 122 generates integrated data by, for example, multiplying each pixel information of the attention map information at each time by each pixel information of the corresponding video. The integrated data generated by the feature integration unit 122 can be said to be video weighted according to the degree of attention of the attention area in time and space. The behavior recognition unit 114 performs recognition processing on this integrated data.

[0070] In the example described above, even in regions where the degree of bit rate reduction is equal to or greater than a predetermined threshold, the weighting of pixel information can be flexibly set without uniformly targeting the region as a masking target. This can further improve the accuracy of the recognition process by the behavior recognition unit 114. Furthermore, even in regions where the degree of bit rate reduction is less than a predetermined threshold as a result of learning, the attention map generation unit 121 can set the weight of the pixel information of the region to a value greater than 0 and less than 1, rather than necessarily setting the weight to "1". This prevents the attention map generation unit 121 from targeting such regions in the recognition process by the behavior recognition unit 114. This can enable the recognition process to be performed efficiently. For example, as a result of learning, the attention map generation unit 121 can set the weight of each pixel information based on information on the degree of spatiotemporal bit rate fluctuation in the QP map sequence.

[0071] In (2B), the attention map generation unit 121 may be a neural network or another type of AI model that has been trained in advance. Furthermore, the attention map generation unit 121 may set regions where the weight of pixel information is a value other than "0" or "1" by rule-based judgment instead of an AI model. For example, two types of judgment thresholds may be set, and for regions where the degree of decrease in bit rate from the reference value is equal to or greater than a first threshold Th1 and less than a second threshold Th2 (Th2>Th1), the weight of each pixel information in the region may be set to a value greater than 0 and less than 1. Three or more types of thresholds may also be set. In this way, the attention map generation unit 121 may use any method to gradually determine the weight of pixel information based on the degree of decrease in bit rate from the reference value.

[0072] (2C) In (2A) and (2B), it is the video that is integrated with the attention map information in the feature integration unit 122. However, the feature integration unit 122 may generate integrated data that integrates the attention map information with video feature information that indicates the spatiotemporal features of the video.

[0073] Fig. 8 is a block diagram showing another example of a compressed information integration unit. In compressed information integration unit 113 shown in Fig. 8, feature information generation unit 120 further includes video feature extraction unit 123 in addition to attention map generation unit 121. Each unit will be described below.

[0074] As shown in (2A), the attention map generation unit 121 generates attention map information (image quality feature information) indicating spatiotemporal features using QP map information indicating the image quality of the video. The attention map generation unit 121 outputs the attention map information to the feature integration unit 122.

[0075] Here, as shown in (2B), the attention map generating unit 121 may be a neural network that has been trained in advance. The training of this neural network is as explained in (2B), so the explanation is omitted here.

[0076] Video feature extraction unit 123 generates video feature information indicating the features of an image for each frame at each time of the video, and outputs the video feature information to feature integration unit 122. The video feature information can be expressed as, for example, a feature amount matrix.

[0077] In this example, the video feature extraction unit 123 is a neural network that has been trained in advance. When training this neural network, a sample video containing a plurality of sample videos is input as a video to the center server 104. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 execute the above-described processes on the acquired sample video. At this time, the video feature extraction unit 123 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct answer label of the behavior recognition corresponding to the sample video is equal to or less than a predetermined threshold. For example, the loss function may be trained so as to have the smallest value among the values ​​that the function can take. The loss function is, for example, cross-entropy loss or mean squared error, but is not limited to these.

[0078] The feature integration unit 122 generates integrated data by integrating the attention map information and the video feature information. The feature integration unit 122 may generate the integrated data by, for example, adding each pixel information of the attention map information at each time and each pixel information of the corresponding video feature information. This allows features in the image in time and space to be emphasized as feature amounts and reflected in the integrated data. However, the feature integration unit 122 may generate the integrated data by processing other than addition. The feature integration unit 122 outputs the generated integrated data to the behavior recognition unit 114.

[0079] As another example, the feature integration unit 122 may be realized by a pre-trained AI model rather than by rule-based processing. For example, the feature integration unit 122 may be realized by a neural network. When this neural network is trained, sample videos including multiple sample videos are input to the center server 104 as videos. The video acquisition unit 111 to the behavior recognition unit 114 of the center server 104 perform the above-described processing on the acquired sample videos. At this time, the feature integration unit 122 is trained so that a loss function calculated based on the recognition result of the behavior recognition unit 114 and the correct answer label of the behavior recognition corresponding to the sample videos is equal to or less than a predetermined threshold. For example, the loss function may be trained to have the minimum value among the values ​​that the function can take. The loss function may be, for example, cross-entropy loss or mean squared error, but is not limited to these.

[0080] With the configuration described above, the behavior recognition unit 114 performs recognition processing on integrated data in which attention map information and video feature information are integrated. At this time, since the video feature information is already indicated in the integrated data, there is no need for the behavior recognition unit 114 to perform processing to extract image features. This allows for simplification of the function of the behavior recognition unit 114.

[0081] Furthermore, the video feature extraction unit 123 that generates the video feature information can be configured with a trained neural network. This makes it possible to accurately capture features in the video, thereby improving the accuracy of behavior recognition by the behavior recognition unit 114.

[0082] In (2C), the image feature extraction unit 123 may be a different type of AI model that has been trained in advance, rather than a neural network. Also, the image feature extraction unit 123 may generate image feature information that indicates the features of an image for each frame by making a rule-based determination.

[0083] The technical concept of the present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0084] For example, in the second embodiment, at least one of lightness information and luminance information in the video may be used instead of or in addition to the QP map information. In areas of the video where the brightness is higher than a predetermined threshold, the accuracy of video recognition may be reduced. Therefore, by generating image quality feature information using lightness information or luminance information and performing recognition processing on integrated data that reflects the image quality feature information, it is possible to suppress the influence of high-brightness areas in the recognition processing even when the video includes high-brightness areas.

[0085] In (2A) and (2B), the weight of each pixel information in the attention map information generated by the attention map generating unit 121 is a value between 0 and 1. However, the value that the weight of each pixel information can take is not limited to this. For example, the weight of each pixel information may be set to a value between 0 and an arbitrary positive number, or may be set to take a negative value.

[0086] In the MEC server 103, information on the bit rate allocated to each terminal 101 may be transmitted from the MEC server 103 to the center server 104. The attention map generation unit 121 may change parameters for generating attention map information for video transmitted from each terminal 101 based on the bit rate. For example, as shown in (2A) and (2B), when the attention map generation unit 121 determines whether there is an area where the degree of decrease from the reference value of the bit rate is equal to or greater than a predetermined threshold, the attention map generation unit 121 can change at least one of the reference value or the threshold in accordance with a change in the bit rate. As an example, when the bit rate allocated to the terminal 101A decreases, the attention map generation unit 121 may lower the reference value and the threshold for the above determination regarding the video of the terminal 101A. In this way, the attention map generation unit 121 can make a determination for each terminal 101 taking into account the bit rate of the entire video and generate a highly accurate attention map. Therefore, the behavior recognition unit 114 can perform recognition processing with high accuracy.

[0087] The center server 104 may output alert information based on the recognition result of the behavior recognition unit 114. For example, when the behavior recognition unit 114 determines that a person in the video is performing a predetermined behavior, the center server 104 can present the alert information on an interface such as a screen. The center server 104 can also display a GUI (Graphical User Interface) on the screen of its display unit and display the video acquired from the terminal 101, the recognition result of the behavior recognition unit 114, an alert, etc. on the GUI.

[0088] In the second embodiment, the compressed information integration unit 113 and the behavior recognition unit 114 are provided in a single device, the center server 104. However, any part of the processing of the compressed information integration unit 113 and the behavior recognition unit 114 may be executed by another device, not by the center server 104. In other words, as described in (1B) of the first embodiment, the processing of the compressed information integration unit 113 and the behavior recognition unit 114 may be realized as a system distributed among multiple devices.

[0089] In the above-described embodiments, this disclosure has been described as a hardware configuration, but this disclosure is not limited to this. This disclosure can also be realized by having a processor in a computer execute a computer program to perform the processes (steps) of the video processing device, the device in the video processing system, or the center server described in the above-described embodiments.

[0090] 9 is a block diagram showing an example of the hardware configuration of an information processing device that executes the processes of the above-described embodiments. Referring to FIG. 9, the information processing device 90 includes a signal processing circuit 91, a processor 92, and a memory 93.

[0091] The signal processing circuit 91 is a circuit for processing signals in accordance with the control of the processor 92. The signal processing circuit 91 may include a communication circuit for receiving signals from a transmitting device.

[0092] The processor 92 is connected (coupled) to the memory 93, and performs the processing of the device described in the above embodiment by reading and executing software (computer programs) from the memory 93. As an example of the processor 92, one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit) may be used, or a plurality of these may be used in parallel.

[0093] The memory 93 is configured with a volatile memory, a nonvolatile memory, or a combination thereof. The memory 93 is not limited to one, and a plurality of memories may be provided. The volatile memory may be, for example, a RAM (Random Access Memory) such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The nonvolatile memory may be, for example, a PROM (Programmable Read Only Memory). Read The memory may be a ROM (Read Only Memory) such as an Erasable Programmable Read Only Memory (EPROM), a flash memory, or a Solid State Drive (SSD).

[0094] The memory 93 is used to store one or more instructions. Here, the one or more instructions are stored as a group of software modules in the memory 93. The processor 92 can perform the processes described in the above embodiments by reading and executing the group of software modules from the memory 93.

[0095] The memory 93 may include memory built into the processor 92 in addition to memory provided outside the processor 92. The memory 93 may also include storage located away from the processors constituting the processor 92. In this case, the processor 92 can access the memory 93 via an I / O (Input / Output) interface.

[0096] As described above, one or more processors included in each device in the above-described embodiments execute one or more programs including instructions for causing a computer to execute the algorithms described using the drawings. This processing enables the information processing described in each embodiment to be realized.

[0097] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0098] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video; an integration unit that generates integrated data by integrating information about the video including spatiotemporal features of the video and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data; A video processing device comprising: (Appendix 2) the feature information generation unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; the integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information; 2. The video processing device according to claim 1. (Appendix 3) the feature information generation unit generates the image quality feature information indicating a map of feature amounts in time and space of the image quality information; the integrating unit generates the integrated data by integrating the image quality characteristic information and image characteristic information, which is information about the image and indicates spatiotemporal characteristics of the image. 2. The video processing device according to claim 1. (Appendix 4) the feature information generation unit further generates the video feature information based on the video. 4. The video processing device according to claim 3. (Appendix 5) the feature information generation unit has a neural network that has been trained so that, when a sample video serving as a sample is acquired as the video, a loss function calculated based on the recognition result of the recognition unit and a correct answer label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. 5. A video processing device according to any one of claims 1 to 4. (Appendix 6) The image quality information is information indicating a compression degree of a region of a frame included in the video. 6. A video processing device according to any one of claims 1 to 5. (Appendix 7) The recognition unit recognizes the behavior of the subject. 7. A video processing device according to any one of claims 1 to 6. (Appendix 8) a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video; an integration unit that generates integrated data by integrating information about the video including spatiotemporal features of the video and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data; A video processing system comprising: (Appendix 9) the feature information generation unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; the integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information; 9. The video processing system of claim 8. (Appendix 10) the feature information generation unit generates the image quality feature information indicating a map of feature amounts in time and space of the image quality information; the integrating unit generates the integrated data by integrating the image quality characteristic information and image characteristic information, which is information about the image and indicates spatiotemporal characteristics of the image. 9. The video processing system of claim 8. (Appendix 11) the feature information generation unit further generates the video feature information based on the video. 11. The video processing system of claim 10. (Appendix 12) the feature information generation unit has a neural network that has been trained so that, when a sample video serving as a sample is acquired as the video, a loss function calculated based on the recognition result of the recognition unit and a correct answer label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. 12. A video processing system according to any one of claims 8 to 11. (Appendix 13) The image quality information is information indicating a compression degree of a region of a frame included in the video. 13. A video processing system according to any one of claims 8 to 12. (Appendix 14) The recognition unit recognizes the behavior of the subject. 14. A video processing system according to any one of claims 8 to 13. (Appendix 15) generating image quality characteristic information indicating spatiotemporal characteristics of image quality information indicating image quality of the video; generating integrated data by integrating information about the video, including spatiotemporal characteristics of the video, and the image quality characteristic information; and performing a recognition process for a subject included in the video based on the integrated data. A computer-implemented image processing method. (Appendix 16) generating the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; generating, as the integrated data, an image in which pixels of the image frame are weighted based on the image quality characteristic information; 16. The video processing method of claim 15. (Appendix 17) generating the image quality feature information indicating a map of the feature amount in space-time of the image quality information; generating the integrated data by integrating the image quality characteristic information and image characteristic information relating to the image, the image characteristic information indicating spatiotemporal characteristics of the image; 16. The video processing method of claim 15. (Appendix 18) generating the video feature information based on the video; 18. The video processing method of claim 17. (Appendix 19) When a sample video serving as a sample is input as the video, learning is performed so that a loss function calculated based on the recognition result of the recognition processing and a correct answer label of action recognition corresponding to the sample video is equal to or less than a predetermined threshold. 19. A video processing method according to any one of appendices 15 to 18. (Appendix 20) The image quality information is information indicating a compression degree of a region of a frame included in the video. 20. A video processing method according to any one of Supplementary Notes 15 to 19. (Appendix 21) The recognition process recognizes the behavior of the subject. 21. A video processing method according to any one of Supplementary Notes 15 to 20. (Appendix 22) generating image quality characteristic information indicating spatiotemporal characteristics of image quality information indicating image quality of the video; generating integrated data by integrating information about the video, including spatiotemporal characteristics of the video, and the image quality characteristic information; and performing a recognition process for a subject included in the video based on the integrated data. A non-transitory computer-readable medium that stores a program that causes a computer to execute a program.

[0099] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the disclosure. [Explanation of symbols]

[0100] 10. Video Processing Device 11 Feature information generation unit 12 Integration unit 13 Recognition part 20. Video Processing System 21 Feature information generation device 22 Recognition device 100 Image Recognition System 101 terminal 102 base station 103 MEC Server 104 Center Server 111 Image acquisition unit 112 QP map information acquisition unit 113 Compressed information integration unit 114 Action recognition unit 120 Feature information generation unit 121 Attention map generation unit 122 Feature integration unit 123 Video feature extraction unit

Claims

1. a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video; an integration unit that generates integrated data by integrating information about the video including spatiotemporal features of the video and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data, the feature information generation unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; the integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information; Image processing device.

2. A video processing system having a center server and an edge server, The center server a feature information generating unit that generates image quality feature information that indicates spatiotemporal features of image quality information that indicates the image quality of the video; an integration unit that generates integrated data by integrating information about the video including spatiotemporal features of the video and the image quality feature information generated by the feature information generation unit; a recognition unit that executes a recognition process for a subject included in the video based on the integrated data, the feature information generation unit generates the image quality feature information indicating a weight of pixel information in the frame of the video based on the image quality information; the integration unit generates, as the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information; The edge server A bit rate of the video to be allocated to the terminal is determined based on a communication environment between the terminal and the video, and information on the determined bit rate of the video is distributed to the terminal. Video processing system.

3. generating image quality characteristic information indicating spatiotemporal characteristics of image quality information indicating image quality of the video; generating integrated data by integrating information about the video, including spatiotemporal characteristics of the video, and the image quality characteristic information; Based on the integrated data, a recognition process is performed on a subject included in the video; generating the image quality characteristic information based on the image quality information, the image quality characteristic information indicating a weight of pixel information in the frame of the video; In generating the integrated data, an image in which pixels of the frame of the image are weighted based on the image quality characteristic information is generated as the integrated data. A video processing program that makes a computer do this.

Images