Reference image cache memory, data request method for prefetch, and data request program for prefetch

Abstract

To improve a cache hit rate and reduce a data transfer amount in a compatible manner.SOLUTION: In a case where a cache error occurs in encoding processing of image data read out of a cache memory section, a reference image cache memory replaces part of image data stored in the cache memory section with image data stored in an external memory. The reference image cache memory comprises a data request section for prefetch by which, on the basis of a motion vector detected in an encoded vicinity region in vicinity regions of an encoding target block, a range of a novel prefetch region in a second direction orthogonal with a first direction, in which the encoding processing is advanced, is determined and image data on the determined novel prefetch region are requested to the external memory.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The disclosed technology relates to a reference image cache memory, a prefetch data request method, and a prefetch data request program. [Background technology]

[0002] Video coding technologies include many video coding methods such as MPEG (Moving Picture Experts Group)-2, MPEG-4, MPEG-4 / AVC, and HEVC (High Efficiency Video Coding). Recently, the latest video coding standard, VVC (Versatile Video Coding), has become more widespread. VVC achieves approximately twice the coding efficiency of the conventional coding method, HEVC. As coding efficiency improves, video with high spatial resolution such as 4K (3840 pixels x 2160 lines) and 8K (7680 pixels x 4320 lines) is now also being handled as a coding process.

[0003] Coding standards such as VVC and HEVC aim to improve coding efficiency by performing interframe coding, which searches, estimates, and compensates for motion between other frames. In interframe coding, a search module searches for coordinates that minimize the cumulative difference value on a pixel-by-pixel basis between an input image and a reference image (an encoded image of a specific frame) to determine a motion vector. In a video coding device, a motion compensation process is performed to generate an image of the block currently being coded using the reference image to which the motion vector is directed according to the determined motion vector.

[0004] The reference images required for motion estimation and motion compensation are a group of previously coded images, and therefore the total data size is very large. For this reason, in the case of a video coding LSI (Large-Scale Integrated circuit), it is not possible to store all the reference image data inside the LSI, and the data must be stored in an external main memory. On the other hand, the external main memory is inferior in high-speed operation performance compared to the LSI calculation processing, so the access latency at the time of data request can affect the coding calculation performance. For this reason, in order to fill the performance gap between the LSI and the external main memory, it is considered to implement a small-capacity temporary storage device capable of high-speed operation, called a cache memory, inside the LSI.

[0005] By implementing a cache memory dedicated to reference image data inside an LSI, it becomes possible to hold reference image data that can be reused in the future in the cache memory in advance, and to conceal the latency of memory access. When each functional block such as a motion search process or a motion compensation process requests reference image data, it is determined whether the requested data is in the cache memory for reference images. In general, when the requested data is in the cache memory for reference images, it is called a hit or cache hit, and when the requested data is not in the cache memory for reference images, it is called a miss or cache miss. In the case of a hit, the requested data can be transferred directly from the cache memory, so there is no need to access the external main memory, and the access latency is concealed.

[0006] Various methods have been proposed for configuring cache memory, but the most representative is a set associative cache that uses the LRU (Least Recently Used) algorithm. A set associative cache is a method in which when a data block (line) in the cache memory is replaced with new data in the event of a cache miss, the replacement candidates are limited to only n candidates in the cache, and is called an n-way set associative cache. LRU is an algorithm that replaces the one that has been accessed the longest among the n replacement candidates.

[0007] In this cache memory configuration, a request signal for new data is sent to the main memory only when there is a miss, and the existing line is replaced according to the above algorithm. The LRU algorithm reflects the temporal locality of data in the cache memory. Temporal locality is the property that a resource referenced at a certain point in time is likely to be referenced again in the near future. By setting the line size to a constant size, data with addresses close to the accessed data are also stored in the cache at the same time, reflecting the locality of data in the address space.

[0008] In this way, a typical cache configuration method can reflect the characteristics that data utilization has locality in time and locality in address space (for example, see Non-Patent Document 1).

[0009] On the other hand, in the case of a cache memory in which only reference image data is stored, in addition to general temporal locality and locality in address space, geometric spatial locality based on coordinate positions within a frame, i.e., two-dimensional locality, is very high. Geometric spatial locality is a property in which when a certain resource is referenced, there is a high possibility that nearby resources will also be referenced.

[0010] The closer the pixel data is to the processing block to be coded in a two-dimensional plane, the easier it is to use it for motion search and motion compensation. Therefore, the distance and position from the own block are highly correlated with the data utilization rate. It is desirable for the reference image data cache memory to have a configuration that reflects the geometric spatial locality, but the configuration of a general cache memory cannot reflect the geometric spatial locality. For example, the proximity in the address space does not match the geometric proximity as the coordinate position in the frame. Therefore, even if the line size is expanded as in a normal cache memory, it can only reflect the locality in the address space, but cannot reflect the geometric spatial locality. In other words, in the configuration of a general cache memory, the closer the two-dimensional distance is, the easier it is to exist in the cache memory.

[0011] For this reason, in addition to retrieving data from an external main memory to a cache memory when a miss occurs, a method has been devised in which data in the vicinity of a block to be coded is pre-fetched from the main memory to the cache memory in advance. This allows pixel data that is geometrically close to the block in question to be preferentially stored in the cache memory in advance, making it easier to hit the neighboring pixel data. This allows the cache memory to take into account the geometric spatial locality of image data, and is expected to improve the hit rate and the throughput of the video coding LSI associated with the improved cache hit rate (see, for example, Patent Document 1). [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] “Cache memory”, [online], [searched on July 18, 2023], Internet <URL:https: / / ja.wikipedia.org / wiki / %E3%82%AD%E3%83%A3%E3%83%83%E3%82%B7%E3%83%A5%E3%83%A1%E3%83%A2%E3%83%AA> [Patent documents]

[0013] [Patent Document 1] Patent Publication No. 2022-21062 Summary of the Invention [Problem to be solved by the invention]

[0014] In the technology described in Patent Document 1, in the motion search and motion compensation processing of video coding, areas with high accessibility are prefetched under the assumption that they are always rectangular of the same size, but in reality, the access range often differs depending on the video or area. For example, in a video or area with a lot of movement, the movement cannot be grasped unless a wide search is performed in a two-dimensional plane, but in a video with little movement, it is sufficient to access only the very nearby area in a two-dimensional plane.

[0015] The purpose of prefetching is to reduce the miss rate, reduce the waiting time when data is transferred from the external main memory after a cache miss, and improve throughput by moving and storing data that is considered to be likely to be used in the future from the external main memory in advance. Therefore, if an area larger than an area that is likely to be accessed is prefetched, a large amount of useless data that will not be used in the future will be prefetched. Therefore, the amount of data to be transferred for prefetching increases more than necessary, which increases the transfer time and reduces the throughput. In addition, the data for prefetching occupies a large proportion of the cache memory. Therefore, as with a normal cache memory, the proportion of the area for storing the most recently used data is reduced. Therefore, when data in an area other than the prefetch area is required, the probability of a cache hit decreases, and the throughput may decrease.

[0016] On the other hand, if an area smaller than the area that is actually more likely to be accessed is prefetched, data that is likely to be used in the future will not be present in the cache memory, which can result in frequent cache misses and reduced throughput due to waiting for transfers.

[0017] The disclosed technology has been developed in consideration of the above points, and aims to provide a reference image cache memory, a prefetch data request method, and a prefetch data request program that can achieve both an improvement in cache hit rate and a reduction in data transfer volume. [Means for solving the problem]

[0018] A first aspect of the present disclosure is a reference image cache memory that replaces a portion of image data stored in a cache memory unit with image data saved in an external memory when a cache miss occurs during an encoding process of image data read from the cache memory unit, and is equipped with a prefetch data request unit that determines the range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds based on a motion vector detected in an already-encoded neighboring area of ​​a block to be encoded, and requests image data of the determined new prefetch area from the external memory.

[0019] A second aspect of the present disclosure is a method for requesting data for prefetching in a reference image cache memory, which replaces a portion of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs during an encoding process of image data read from the cache memory unit, and performs a process including: a computer determines the range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requests image data of the determined new prefetch area from the external memory.

[0020] A third aspect of the present disclosure is a prefetch data request program in a reference image cache memory that replaces a portion of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs during an encoding process of image data read from the cache memory unit, and causes a computer to perform a process including determining the range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requesting image data of the determined new prefetch area from the external memory. Effect of the Invention

[0021] According to the disclosed technique, it is possible to improve the cache hit rate while reducing the amount of data transfer. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram showing the internal configuration of a video encoding device. [Diagram 2] 11 is a block diagram illustrating a functional configuration of an external memory data request unit. FIG. [Diagram 3] FIG. 13 is a diagram for explaining an encoding processing area. [Figure 4] FIG. 11 is a diagram for explaining a reference region. [Diagram 5] FIG. 11 is a diagram for explaining a reference region. [Figure 6] FIG. 11 is a diagram for explaining a reference region. [Figure 7] FIG. 11 is a diagram for explaining a reference region. [Figure 8] FIG. 11 is a diagram for explaining a reference region. [Figure 9] FIG. 11 is a diagram for explaining a reference region. [Figure 10] FIG. 13 is a diagram for explaining a new prefetch area. [Figure 11]FIG. 13 is a diagram for explaining determination of a new prefetch area. [Figure 12] 11 is a block diagram illustrating a functional configuration of a prefetch data request unit. FIG. [Figure 13] FIG. 13 is a diagram for explaining determination of a new prefetch area. [Figure 14] FIG. 13 is a diagram for explaining determination of a new prefetch area. [Figure 15] 11 is a block diagram illustrating a hardware configuration of a prefetch data request unit. FIG. [Figure 16] 13 is a flowchart of a prefetch data request process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0024] 1 is a block diagram showing the internal configuration of a video encoding device 10 according to the present embodiment. In the following, the term "encoding block" refers to a macroblock in the MPEG-2 and H.264 / AVC standards, and refers to a coding unit (CU) or a prediction unit (PU) in the HEVC and VVC standards.

[0025] The video encoding device 10 includes a video encoding LSI 50 and an external memory 900. The video encoding LSI 50 and the external memory 900 are connected via a bus.

[0026] The video encoding LSI 50 performs encoding processing on the original video signal to be encoded in accordance with the encoding standard.

[0027] The external memory 900 stores various data such as reference image data used in the video encoding LSI 50. Address information of the various data in the external memory 900 is shared with the video encoding LSI 50.

[0028] The video encoding LSI 50 includes an encoding calculation unit 100 and a reference image data cache memory 200 .

[0029] The encoding calculation unit 100 performs encoding calculation processes such as motion search and motion compensation. The encoding calculation unit 100 is composed of multiple cores (core_1, . . . , core_M) that operate in parallel. M is an integer equal to or greater than 2. Each core is a functional unit that performs encoding calculation processes such as at least a pre-motion search processing unit, a motion search processing unit, a merge search processing unit, and a motion compensation processing unit.

[0030] The encoding calculation process of each core is pipelined. In the case of a core that requires reference image data during processing (hereinafter referred to as a “reference image using core”), the reference image using core is connected to a reference image data cache memory 200.

[0031] Each reference image use core converts information such as the reference frame number, the designation of the reference color space YUV, and the position coordinates within the reference frame into address information for the required reference image data, and issues a data request to the reference image data cache memory 200. The encoding calculation unit 100 sequentially converts information transmitted in parallel from each reference image use core, and outputs address information of the requested data (hereinafter referred to as "request address information") in order to the reference image data cache memory 200. At the start of each stage of pipeline processing, the encoding calculation unit 100 transmits a pipeline update signal indicating a stage update and a motion vector (encoding result data) required for prefetching to the reference image data cache memory 200.

[0032] The reference image data cache memory 200 is an n-way set associative cache memory. The reference image data cache memory 200 includes a cache memory unit 300, a data determination unit 400, a data replacement control unit 500, and an external memory data request unit 600. The reference image data cache memory 200 judges whether the requested data exists in the cache memory unit 300 according to the request address information requested by the encoding calculation unit 100. The request address information is separated into a tag of the upper bits and an index of the lower bits. The index indicates in which set in the cache memory unit 300 the requested data may exist. The tag is used as a key to search whether the requested data exists in a certain set (n line data in the case of n-way set associative) indicated by the index.

[0033] The cache memory unit 300 is composed of a data array 310 and a tag array 320. The data array 310 stores actual line data. This line data is information indicating a reference area referenced by a block to be coded. The tag array 320 stores tag data corresponding to each line data in the data array 310. An index is used to specify a specific line in the data array 310 and the tag array 320. The index is converted by an address decoder and used to specify an address in the data array 310 and the tag array 320. In the case of an n-way, there are n lines with the same index address.

[0034] The data determination unit 400 is composed of a data selection unit 410 and a tag comparison unit 420. The tag comparison unit 420 compares the n tags read from the tag array 320 with the tag of the requested address information, and outputs comparison result information indicating which way in each of the n sets was hit or all were missed to the data selection unit 410, the data replacement control unit 500, and the external memory data request unit 600. Specifically, for example, in the case of N=4, i.e., 4-way, the tag comparison unit 420 reads four tags from the tag array 320 and compares whether a tag identical to the tag of the requested address information exists. If a tag identical to the tag of the requested address information exists, the tag comparison unit 420 outputs the way number of the tag identical to the tag of the requested address information to the data selection unit 410 as a cache hit.

[0035] When a cache hit occurs, the data selection unit 410 reads the line data corresponding to the way number output from the tag comparison unit 420 from the data array 310, and outputs it to the encoding calculation unit 100. The line data output from the data selection unit 410 is transmitted to the core that made the request.

[0036] When a cache miss occurs in the encoding process of the image data read from the cache memory unit 300, the data replacement control unit 500 replaces a part of the image data stored in the cache memory unit 300 with the image data stored in the external memory 900. Specifically, when a cache miss occurs and the corresponding requested data is transferred from the external memory 900, the data replacement control unit 500 determines which line data in the data array 310 is to be replaced with the requested data. In the case of n-way, the data replacement control unit 500 determines which way in each of the n sets is to be replaced. In the case of the LRU algorithm, history information indicating the order in which the n line data of each set was used is stored in the internal data of the data replacement control unit 500 for the number of sets. Therefore, the data replacement control unit 500 can recognize which line data was used most recently and which line data was used least recently among the n line data of each set.

[0037] When a hit occurs, the data replacement control unit 500 receives from the tag comparison unit 420 comparison result information indicating which way was hit, and also receives the index of the requested address information, thereby updating the internal data. When a miss occurs, the data replacement control unit 500 determines the way with the oldest value in the order of usage history among the n ways in the set as the way number to be replaced. In this way, when a cache miss occurs, the data replacement control unit 500 replaces a part of the cache stored in the cache memory unit 300 with data saved in the external memory 900.

[0038] The external memory data request unit 600 makes a data request to the external memory 900. The request address information input from the encoding calculation unit 100 and the comparison result information input from the tag comparison unit 420 are received as input.

[0039] The external memory data request unit 600 also receives as input a pipeline update signal indicating a pipeline stage update and a prefetch motion vector as a data request for prefetch. The external memory data request unit 600 determines final request address information in consideration of both the cache miss data request and the prefetch data request, and outputs the request address information to the external memory 900 together with the data request signal.

[0040] The external memory 900 outputs the image data corresponding to the request address information to the reference image data cache memory 200. As a result, one line of data in the data array 310 is overwritten.

[0041] Furthermore, in addition to requesting data from the external memory 900 when a cache miss occurs, the external memory data request unit 600 pre-fetches image data in the vicinity of the coding block to be coded (hereinafter, the coding target block) in advance to the cache memory unit 300. The image data in the vicinity of the coding target block is image data of a pixel area that is likely to be referenced in image space as the process proceeds. This makes it possible to configure a cache memory that takes into account the geometric spatial locality of image data and improve the cache hit rate. Details will be described below.

[0042] FIG. 2 is a block diagram showing the functional configuration of the external memory data request unit 600 in this embodiment.

[0043] The external memory data request unit 600 includes a cache miss data request unit 610, a prefetch data request unit 620, and a data request adjustment unit 630. In the external memory data request unit 600, data requests are made in parallel from the cache miss data request unit 610 and the prefetch data request unit 620.

[0044] The cache miss data request unit 610 is a functional unit that issues a data transmission request in the same manner as in the conventional method. The cache miss data request unit 610 refers to the comparison result information, and issues a data transmission request to the external memory 900 only when a cache miss occurs. When a cache miss occurs, the cache miss data request unit 610 uses the request address information input from the encoding calculation unit 100 as the cache miss request address information, and generates a data transmission request including the cache miss request address information and a data request instruction. The cache miss data request unit 610 outputs the generated data transmission request to the data request adjustment unit 630.

[0045] The prefetch data request unit 620 determines the range of a new prefetch area in a second direction perpendicular to the first direction in which the encoding process progresses based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requests image data of the determined new prefetch area from the external memory 900.

[0046] The prefetch data request unit 620 will now be described in detail.

[0047] In order to prefetch image data in the vicinity of the block to be coded, the prefetch data request unit 620 generates a data transmission request including prefetch request address information and a data request instruction, using request address information corresponding to the image data in the vicinity of the block to be coded as prefetch request address information. The prefetch data request unit 620 outputs the generated data transmission request to the data request adjustment unit 630.

[0048] Specifically, the prefetch data request unit 620 first calculates the intraframe coordinate position of the encoding target block currently being encoded by each core of the encoding calculation unit 100 based on the pipeline update signal. Next, the prefetch data request unit 620 determines the position and size of a prefetch area to be newly prefetched based on the calculated coordinate position of the encoding target block and the motion vector, and calculates the address of the determined prefetch area. Then, the prefetch data request unit 620 generates a data transmission request including prefetch request address information and a data request instruction, using the calculated address of the prefetch area as prefetch request address information. As a result, the prefetch data request unit 620 makes a data transmission request for prefetch.

[0049] The data request adjustment unit 630 inputs the data transmission request output from the cache miss data request unit 610 and the data transmission request output from the prefetch data request unit 620. The data request adjustment unit 630 adjusts the input data transmission requests, and then sequentially requests the external memory 900 for image data according to the request address information.

[0050] Next, specific processing of the prefetch data request unit 620 in this embodiment will be described with reference to FIGS.

[0051] In addition to requesting data from the external memory 900 when a cache miss occurs, the external memory data request unit 600 also requests image data in the vicinity of a block to be coded from the external memory 900 in advance to prefetch the data. Furthermore, the prefetch data request unit 620 shown in Fig. 2 adaptively changes the position and size of the prefetch area based on the motion vector (motion search / motion compensation result), which is coding result data of the surrounding area, thereby improving the throughput of the video coding hardware.

[0052] The process of the prefetch data request unit 620 will now be described in detail.

[0053] 3 shows a positional relationship of the encoding process at a certain timing within a frame 700. In the encoding calculation unit 100, it is assumed that each of M cores from Core_1 to Core_M simultaneously performs encoding-related processes.

[0054] The encoding arithmetic processing of each core is pipelined, and the unit of the pipeline stage is a coding tree unit (CTU) as an example of an encoding block. The encoding arithmetic processing is performed sequentially in the x direction (first direction) from the CTU in the first line at the top left of Fig. 3, and when the encoding arithmetic processing of one line of CTUs is completed, the encoding arithmetic processing is sequentially performed in the x direction from the CTU at the left end of the second line. The encoding arithmetic processing is repeated until the encoding arithmetic processing of the CTU in the lower right is completed.

[0055] If the pipeline progresses in the order Core_1, Core_2, ..., Core_M, let x be the index of the CTU being processed by Core_1, which is in charge of the first pipeline stage. In this case, as shown in Figure 3, at the same time, the index of the CTU being processed by Core_2 is (x+1), the index of the CTU being processed by Core_(M-1) is (x+M-2), and the index of the CTU being processed by Core_M is (x+M-1). Let P be the stage number of the pipeline stage at this time. If the size of the CTU is S (pixels) x S (pixels), the total processing area is M x S (pixels) x S (pixels).

[0056] In a video coding process, reference image data that is geometrically close to a coding target block tends to have a high data availability and a high data reference frequency. Furthermore, due to factors such as the search range setting in a motion estimation process, the availability and reference frequency do not decrease gradually as the geometric distance from the coding target block increases, but may change drastically depending on whether the data is within a certain distance from the coding target block.

[0057] If a highly referenceable neighboring region is a range with a length of H (pixels) or less in the up-down direction, i.e., the y direction (second direction), and a length of W (pixels) or less in the left-right direction, i.e., the x direction, centered on the CTU to be coded as the block to be coded, then Fig. 4 shows a highly referenceable neighboring region (hatched region) 710 in a CTU at position (x+M-1) during pipeline stage P. Fig. 5 shows a highly referenceable neighboring region 712 in a CTU at position x during pipeline stage P. Note that, hereinafter, the rightward direction is the positive x-direction, the leftward direction is the negative x-direction, the downward direction is the positive y-direction, and the upward direction is the negative y-direction.

[0058] 6 shows a neighborhood region 714 obtained by summing up the neighborhood regions with high reference probability in all CTUs from position x to position (x+M-1) during pipeline stage P. The number of pixels PN in the neighborhood region 714 shown in FIG. 6 is expressed by the following equation.

[0059] PN=((M-1)×S+2W)×2H ···(1)

[0060] In the examples of Figs. 4 to 6, it is assumed that the neighboring regions with high accessibility in the motion search process and motion compensation process in video coding are always rectangular with the same size. However, in reality, the access range often differs depending on the content of the video. For example, in the case of a video with a lot of movement, the movement cannot be grasped unless a wide search is performed in a two-dimensional plane, but in the case of a video with a small movement, it is sufficient to access only the very nearby in a two-dimensional plane. As an example, Fig. 7 shows a neighboring region 716 with high accessibility in a CTU at a position x during a pipeline stage P. In the example of Fig. 7, the length from the center of the CTU 800 at the position x in the upward direction is Hu, and the length from the center of the CTU 800 at the position x in the downward direction is Hl. In the example of Fig. 7, the center of the neighboring region 716 is shifted in the y direction from the center of the CTU 800 at the position x, so Hu ≠ Hl. Also, the length in the y direction of the neighboring region 800 is not necessarily the same for all CTUs as in Fig. 6, and in the example of Fig. 7, Hu + Hl ≠ 2 × H.

[0061] In addition, the nearby area with high accessibility may not be a simple rectangle. For example, the left side of the CTU moves a lot, so it is necessary to search a wide area, but the right side of the CTU moves little, so it is sufficient to access only the neighborhood. As an example, FIG. 8 shows a nearby area 718 with high accessibility for a CTU at position x during pipeline stage P. In the example of FIG. 8, the nearby area 718 is not a simple rectangle, but has a shape that includes multiple rectangles. In the example of FIG. 8, the length of the nearby area 718 in the x direction is 3×S=2×W. Therefore, the nearby area 718 is composed of three unit rectangular areas 718A, 718B, and 718C, each of which has a length S in the x direction.

[0062] FIG. 9 shows a neighborhood area 720 obtained by summing up neighborhood areas with high reference probability in all CTUs from position x to position x+M−1 during pipeline stage P.

[0063] In the example of Figure 9, the lengths in the y direction of the multiple unit rectangular regions 722 constituting the nearby region 720 with high referenceability are not the same. The lengths in the x direction of the multiple unit rectangular regions 722 are the same. Thus, in the example of Figure 9, the lengths in the x direction of the unit rectangular regions 722 constituting the nearby region 720 with high referenceability are the same as in Figure 6, but the lengths in the y direction change every S (pixels) along the x direction, so that the nearby region 720 has a more complex shape than in the case of Figure 6.

[0064] The prefetch data request unit 620 prefetches the minimum amount of image data required into the cache memory unit 300 by performing prefetching so that the length in the y direction and the center position of the unit rectangular area 722 change as shown in Fig. 9. This makes it possible to maintain a high cache hit rate while preventing unnecessary image data from being prefetched. This makes it possible to reduce the amount of data transfer and the amount of prefetched image data in the cache memory unit 300.

[0065] FIG. 10 shows an example of prefetching a newly required neighboring area in preparation for the pipeline stage advancing by one (P+1). In the next pipeline stage (P+1), CTU(x-1) will be included as a processing area for the encoding process. Therefore, the neighboring area that will become more likely to be referenced again will also move one CTU to the right. Since the image data of this new area is likely not accessed in the pipeline stage P, in a normal cache memory configuration, it is likely that the image data will not exist in the cache memory unit 300 at the timing of the pipeline stage (P+1). For this reason, there is a possibility that many cache misses will occur after the pipeline stage is updated, which may lead to a decrease in encoding efficiency and a breakdown in real-time processing.

[0066] For this reason, it is necessary to set a new prefetch area 724 as a neighboring area 720 obtained by adding up the neighboring areas that are highly likely to be referenced in the pipeline stage P, in other words, a rectangular area of ​​width S located to the right of the neighboring area 720 that has already been prefetched in the pipeline stage P, and to transfer the new prefetch area 724 from the external memory 900 to the cache memory unit 300 in advance at the time of the pipeline stage P. If the new prefetch area 724 can be reliably stored in the cache memory unit 300 during the pipeline stage P, then at the time of the next pipeline stage (P+1), it will be necessary to access only the cache memory unit 300 without accessing the external memory 900.

[0067] In the example of Fig. 10, as in Fig. 9, 3 × S = 2 × W. Therefore, the new prefetch area is likely to be accessed by the three future CTUs, namely, CTU(x-1) to CTU(x-3). Therefore, by appropriately estimating the size and position in the y direction of the neighboring area that is likely to be referenced by CTU(x-1) to CTU(x-3), it is possible to prefetch only the minimum amount of image data that is necessary.

[0068] The size and position of the new prefetch area are adaptively changed only in the y direction. The size and position of the new prefetch area in the x direction are not changed. This is because, if the size and position in the x direction are changed, gaps and overlaps may occur in the neighboring area 720 in the pipeline stage P, that is, the neighboring area 720 that has already been prefetched when the position of Core_1 is x, as shown in FIG. 10. In the motion search process and motion compensation process in video coding, the prefetched area is considered to be normal when it is stored in the cache memory unit 300 without gaps or overlaps, and the size and position in the x direction are fixed. On the other hand, if there is an area that is clearly considered unnecessary in the motion search process and motion compensation process, the size and position in the x direction may also be adaptively changed to configure an efficient cache memory.

[0069] Next, the coding result data shown in Fig. 1, that is, the data held as a motion vector, will be described with reference to Fig. 11. The prefetch data request unit 620 uses a part of the coding result data to determine the size and position of a new prefetch area.

[0070] FIG. 11 shows an encoding result holding CTU802 (hatched CTU) that holds the encoding result data. With the CTU(x) at the latest position x currently being processed as a reference, encoding result data for 1 CTU line +α is stored. One CTU line is a CTU for the width in the x direction of the frame 700. In the example of FIG. 11, one CTU line is composed of 16 CTUs. α varies depending on the length in the x direction of the nearby area with high reference possibility. In the example of FIG. 11, α=2, that is, 2 CTUs. Therefore, the encoding result holding CTU802 is composed of 18 CTUs. The encoding calculation unit 100 holds each of the motion vectors 804 calculated in each CTU of the encoding result holding CTU802 as encoding result data. For example, when the final encoding result is used, the motion vector finally selected as a result of the motion search process, that is, the motion vector 804 used in the motion compensation process, is stored. Furthermore, when it is desired to use the range previously searched in the motion search process rather than the final encoding result, a characteristic motion vector 804 generated during the motion search process may be stored. For example, when the motion search process is performed for each CTU, the motion vector 804 obtained when the furthest area is searched may be stored for each CTU.

[0071] FIG. 12 shows details of the prefetch data request unit 620 that can adaptively change the position and size of the new prefetch area 724.

[0072] The prefetch data request unit 620 includes a reference point position determination unit 622 , a y-direction information determination unit 624 , a new prefetch area determination unit 626 , and an address information conversion unit 628 .

[0073] The reference point position determination unit 622 determines whether the pipeline stage is updated by acquiring a pipeline update signal, and when the pipeline stage is updated, determines the position of the reference point of the new prefetch area. An example of the position of the reference point of the new prefetch area is shown in Fig. 13. In the example of Fig. 13, a position shifted by 2 x S to the right from the center position of the current CTU(x) is calculated as the reference point 806 of the new prefetch area 724. Here, the position of the reference point 806 is set to (xb, yb).

[0074] The y-direction information determination unit 624 selects only a specific motion vector 804 from among the motion vectors 804 held in the encoding result holding CTU 802, and inputs it to determine the y-direction length and position of the new prefetch area 724. That is, it determines Hu(x) and Hl(x). As shown in FIG. 13, Hu(x) is the length in the upward direction from the y-coordinate yb of the reference point 806, and Hl(x) is the length in the downward direction from the y-coordinate yb of the reference point 806. That is, the y-direction length of the new prefetch area 724 is Hu(x)+Hl(x).

[0075] The motion vector 804 used by the y-direction information determination unit 624 will be described with reference to FIG. 14. Since the new prefetch area 724 is used by CTU(x-1) to (x-3) as shown in FIG. 10, it is important which area CTU(x-1) to (x-3) access in the motion search process and motion compensation process. For this reason, information on CTUs located near CTU(x-1) to (x-3) is used as a reference. The type of motion search process and motion compensation process performed in the past on these CTUs is used as a reference. For example, when 7 CTUs in the x direction and 2 CTUs in the y direction are set as the reference area for CTU(x-2), which is the center of CTU(x-1) to (x-3), the hatched area becomes the reference area 808 as shown in FIG. 14. Since the coding process has not yet been completed for the CTUs to the right and below CTU(x), the coding result data that can be used are the CTUs to the left and above CTU(x-1) to (x-3).

[0076] The nine CTUs shown in Fig. 14 are used to determine the length and position of the new prefetch area 724 in the y direction, and are set as a reference area 808 for determining the new prefetch area. The reference area 808 is always part of the encoding result holding CTU 802, and therefore a motion vector 804 is stored as the encoding result data in Fig. 1. The y direction information determination unit 624 calculates and outputs the length and position of the new prefetch area 724 in the y direction based on the motion vector 804 of the reference area 808.

[0077] Fig. 13 shows an example in which the nine motion vectors 804 shown in Fig. 14 are grouped together with the center of CTU(x-2) as the origin. In this case, the y-direction information determination unit 624 may set Hu(x) and Hl(x) so that the new prefetch region 724 is the smallest range that includes all of the nine motion vectors 804 detected in each of the nine CTUs shown in Fig. 14.

[0078] As a result, for example, when the motion vector 804 finally selected as a result of the motion search process, i.e., the motion vector 804 used in the motion compensation process, is stored, if CTU(x-1)-(x-3) have the same motion as in the past motion search process, it is possible to prefetch all image data required for future motion compensation processes of CTU(x-1)-(x-3). It can also be said that there is a low possibility of prefetching unnecessary image data at that time.

[0079] Also, for example, if the motion vector from the furthest search is retained when motion estimation is performed for each CTU, and CTU(x-1)-(x-3) move in a similar manner to past CTU processing, it will be possible to prefetch all image data required for future motion estimation processing of CTU(x-1)-(x-3). It can also be said that there is a low possibility of prefetching unnecessary image data in this case.

[0080] As described above, by using the characteristic motion vector 804 as past encoding result data to determine Hu(x) and Hl(x), which represent the length and position in the y direction at the time of prefetching, prefetching can be performed without excess, deficiency, or waste.

[0081] In the example of Figure 13, Hu(x) and Hl(x) are set so that the new prefetch area 724 is the smallest range that encompasses all of the motion vectors 804 in the reference area 808, but the method of setting Hu(x) and Hl(x) is not limited to this.

[0082] For example, Hu(x) and Hl(x) may be set so that the smallest range that includes all of the motion vectors 804 in the reference region 808, excluding the maximum motion vector 804 in the downward direction, i.e., the positive y direction, and the maximum motion vector 804 in the upward direction, i.e., the negative y direction, becomes the new prefetch region 724. This is because an extremely large motion vector 804 is considered an outlier and is not required for the current prefetch.

[0083] Also, for example, an area obtained by adding a margin in the y direction to the minimum range that includes all the motion vectors 804 of the reference area 808 may be set as the new prefetch area 724. In other words, an area obtained by expanding the minimum range in the y direction by the margin is set as the new prefetch area 724. This ensures prefetching in the next CTU processing.

[0084] The new prefetch area determination unit 626 determines the new prefetch area 724 based on the position (xb, yb) of the reference point 806 of the new prefetch area 724 and the length and position of the new prefetch area 724 in the y direction.

[0085] Specifically, as shown in FIG. 13, it is a rectangular area S×(Hu(x)+Hl(x)) with its four corners at (xb-s / 2, yb-Hu), (xb+s / 2, yb-Hu), (xb-s / 2, yb+Hl), and (xb+s / 2, yb+Hl).

[0086] The address information conversion unit 628 determines the address number of each pixel data in the new prefetch area 724 and outputs it to the data request adjustment unit 630 as request address information for prefetching.

[0087] As described above, in this embodiment, the prefetch data request unit 620 determines the range of the new prefetch area 724 in the y direction orthogonal to the x direction in which the encoding process proceeds, based on a motion vector detected in an already-encoded neighboring area of ​​the encoding target block, and requests image data of the determined new prefetch area 724 from the external memory 900. This makes it possible to simultaneously improve the cache hit rate and reduce the amount of data transfer.

[0088] The prefetch data request unit 620 in this embodiment may be realized by a computer 60 as shown in FIG.

[0089] The computer 60 includes a central processing unit (CPU) 60A that handles the processing of each functional unit of the prefetch data request unit 620 shown in Fig. 12. The computer 60 also includes a read only memory (ROM) 60B, a random access memory (RAM) 60C, a non-volatile memory 60D, and an input / output interface (I / O) 60E. A prefetch data request program 60P that causes the computer 60 to function as each functional unit of the prefetch data request unit 620 is stored in the non-volatile memory 60D. The CPU 60A, the ROM 60B, the RAM 60C, the non-volatile memory 60D, and the I / O 60E are connected to each other via a bus 60F.

[0090] The nonvolatile memory 60D is an example of a storage device that maintains stored information even if the power supplied to the nonvolatile memory 60D is cut off, and may be, for example, a semiconductor memory. The nonvolatile memory 60D does not necessarily have to be included in the computer 60, and for example, a portable storage device that can be attached to and detached from the computer 60 may be used as the nonvolatile memory 60D.

[0091] The encoding calculation unit 100 and the data request adjustment unit 630 are connected to the I / O 60E.

[0092] The CPU 60A executes the prefetch data request process shown in FIG. 16 by loading the prefetch data request program 60P into the RAM 60C and executing it.

[0093] In step S100, CPU 60A, functioning as reference point position determining unit 622, performs the process of calculating the position (xb, yb) of reference point 806 as described above.

[0094] In step S101, the CPU 60A, functioning as the y direction information determination section 624, performs the process of determining the length and position in the y direction of the new prefetch area 724, as described above.

[0095] In step S102, the CPU 60A, as the new prefetch area determination unit 626, performs processing to determine the new prefetch area 724 based on the position (xb, yb) of the reference point 806 of the new prefetch area 724 and the length and position of the new prefetch area 724 in the y direction, as described above.

[0096] In step S103, the CPU 60A functions as the address information conversion section 628 to determine the address number of each pixel data in the new prefetch area 724, as described above, and outputs it to the data request adjustment section 630 as request address information for prefetch.

[0097] In the above, an example in which the prefetch data request program 60P is stored in the nonvolatile memory 60D has been described, but the storage destination of the prefetch data request program 60P is not limited to the nonvolatile memory 60D. The prefetch data request program 60P of the present disclosure can also be provided in a form recorded in a storage medium readable by the computer 60. For example, the prefetch data request program may be provided in a form recorded in an optical disk such as a CD-ROM (Compact Disk Read Only Memory) or a DVD-ROM (Digital Versatile Disk Read Only Memory). The prefetch data request program 60P may also be provided in a form recorded in a portable semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. The nonvolatile memory 60D, the CD-ROM, the DVD-ROM, the USB, and the memory card are examples of non-transitory storage media.

[0098] Furthermore, the prefetch data request program 60P may be downloaded from an external device, and the downloaded prefetch data request program 60P may be stored in the non-volatile memory 60D. In this case, the prefetch data request unit 620 reads the prefetch data request program 60P downloaded from the external device and executes the prefetch data request process.

[0099] Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0100] The following supplementary notes are further disclosed regarding the above embodiment.

[0101] (Additional note 1) A reference image cache memory that replaces a part of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs in an encoding process of image data read from the cache memory unit, Memory, at least one processor coupled to the memory; Including, The processor, determining a range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process progresses, based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requesting image data of the determined new prefetch area from the external memory; a reference image cache memory configured to:

[0102] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to execute a pre-fetch data request process in a reference image cache memory for replacing a part of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs in an encoding process of image data read from the cache memory unit, The prefetch data request process includes: determining a range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process progresses, based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requesting image data of the determined new prefetch area from the external memory; Non-transitory storage medium. [Explanation of symbols]

[0103] 10 Video Encoding Device 50 Video encoding LSI 60 Computer 60P Prefetch data request program 100 Encoding operation section 200 Reference image data cache memory 300 Cache memory section 400 Data Decision Unit 500 Data replacement control section 600 External memory data request unit 610 Data request unit for cache miss 620 Prefetch data request part 622 Reference point position determination section 624 y-direction information determination unit 626 New Prefetch Area Decision Unit 628 Address Information Conversion Unit 630 Data Request Coordination Department 700 frames 720 Nearby Region 722 Unit Rectangular Area 724 new prefetch areas 800 Neighborhood Area 804 Motion Vectors 806 Reference point 808 Reference area 900 External Memory

Claims

1. A reference image cache memory that replaces a part of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs in an encoding process of image data read from the cache memory unit, a prefetch data request unit that determines a range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds, based on a motion vector detected in an already-encoded neighboring area of ​​the neighboring area of ​​the encoding target block, and requests image data of the determined new prefetch area from the external memory; A reference image cache memory comprising:

2. the neighborhood region includes a plurality of coding blocks; the prefetch data request unit determines, as the new prefetch area, a range including all of the motion vectors detected in each of the plurality of coding blocks; 2. The reference image cache memory of claim 1.

3. the prefetch data request unit determines, as the new prefetch area, a minimum range that includes all of the motion vectors detected in each of the plurality of coding blocks; 3. The reference image cache memory according to claim 2.

4. the prefetch data request unit determines an area obtained by adding a margin in the second direction to the minimum range as the new prefetch area; 4. The reference image cache memory according to claim 3.

5. the neighborhood region includes a plurality of coding blocks; the prefetch data request unit determines, as the new prefetch area, a range including all of the motion vectors detected in each of the plurality of coding blocks, excluding a maximum motion vector in the positive second direction and a maximum motion vector in the negative second direction; 2. The reference image cache memory of claim 1.

6. 1. A prefetch data request method for a reference image cache memory, which replaces a part of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs in an encoding process of image data read from the cache memory unit, comprising: The computer determining a range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds, based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requesting image data of the determined new prefetch area from the external memory; The method of claim 1, further comprising:

7. A prefetch data request program for a reference image cache memory, which replaces a part of image data stored in a cache memory unit with image data stored in an external memory when a cache miss occurs in an encoding process of image data read from the cache memory unit, comprising: On the computer, determining a range of a new prefetch area in a second direction orthogonal to a first direction in which the encoding process proceeds, based on a motion vector detected in an already-encoded neighboring area of ​​the block to be encoded, and requesting image data of the determined new prefetch area from the external memory; A prefetch data requesting program that executes a process including the steps of:

Images