Method for solving multi-process stream extraction by application layer

Abstract

The invention provides a method for solving multi-process stream taking of an application layer. The method comprises the following steps: S1, inter-process communication; s2, designing a queue flow; s3, the host process takes a stream; s4, taking a stream in a stream taking process; and S5, self-testing. The method is realized on the basis that a self-research platform accesses other algorithm company platforms, and application programs of intelligent video processing technologies of other company platforms are simultaneously operated on a self-research chip platform. The stream acquisition process can acquire the code stream to isolate the process and acquire the video source at the same time, so that the client can directly access programs of other algorithm companies to carry out special video processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video processing, and in particular relates to a method for multi-process stream acquisition at an application layer. Background Art

[0002] With the development of computer technology and the widespread application of computer vision principles, the use of computer image processing technology to detect and track targets in real time has become increasingly popular. The application of intelligent video is mainly reflected in intelligent transportation systems, intelligent monitoring systems, and human behavior detection. With the increasing complexity of application scenarios, many customers are no longer able to meet the application scenarios of independent platforms. In order to meet the needs of customers for multi-platform access, multi-process streaming has been developed.

[0003] However, the existing platform video source of Beijing Ingenic Integrated Circuit Co., Ltd. (hereinafter referred to as: Ingenic) can only be used by the current process, and other processes cannot obtain the video source, resulting in the inability to access the platforms of other algorithm companies. Summary of the invention

[0004] In order to solve the above problems, the purpose of this application is to solve the problem of obtaining the video source of the main process to access the video algorithm processing program of other algorithm companies while running the platform program developed by Ingenic. That is, based on the Ingenic platform accessing other algorithm company platforms, the application programs of intelligent video processing technology of other company platforms are simultaneously run on the Ingenic chip platform. The stream acquisition process can obtain the code stream to isolate the process and get the video source at the same time, so that customers can directly access the programs of other algorithm companies for special video processing.

[0005] Specifically, the present invention provides a method for solving multi-process flow acquisition at the application layer, and the method comprises the following steps:

[0006] S1. Inter-process communication:

[0007] Because it is inter-process communication, the fastest shared memory communication method is used to ensure maximum efficiency;

[0008] To obtain streams between processes, multiple processes need to be distinguished. The SDK uses the pid number to distinguish and process. Each process corresponds to the streams of NR_MAX_FS_CHNS channels. This macro indicates the maximum number of channels allowed on the chip SDK. Different chips have different performances, and the number of channels corresponding to this macro is also different.

[0009] The structure shared between multiple processes in this method is based on the frame encapsulated by the SDK. Multiple processes need to compete to obtain the right to use the frame. In order to solve the competition problem, a synchronization mechanism is introduced. For this method of using multiple processes to obtain streams, the synchronization mechanisms used in the application layer include semaphores and mutexes. Semaphores are used for synchronization and counting, and mutexes are used for synchronization.

[0010] In order to reduce the problem of frame loss, the SDK applies for a maximum of nrVBs frames for rotation in the system. In order to ensure synchronization, a maximum of nrVBs frames can be obtained in multi-process streaming, this function is also adapted.

[0011] In order to more conveniently manage the shared frame, semaphore, mutex and other variable information, this method puts these variable information into a gpshmframe structure and uses a queue to manage its content;

[0012] S2. Queue process design, including:

[0013] S2.1. In order to prevent the main process from being affected when the stream fetching process fetches the stream, a non-equal operation is required. After fetching a frame, the main process puts the stream into the queue and closes it until the fetching interface is called again.

[0014] S2.2. Since it is a queue operation between multiple processes, the SDK enters the queue and the stream retrieval process exits the queue, so synchronization needs to be ensured. The ways to ensure synchronization between process communications include synchronization locks and semaphores. The process of using queues includes:

[0015] S2.2.1, Initialize the queue

[0016] S2.2.2, Enqueue Operation:

[0017] Sem_wait

[0018] Lock

[0019] Add data to the queue

[0020] Notify the dequeue operation to fetch the stream (cond / sem_post)

[0021] Unlock

[0022] S2.2.3, Dequeue operation:

[0023] Lock

[0024] Wait until the queue operation notifies the user to fetch the stream

[0025] Retrieve data from the queue

[0026] Unlock

[0027] Sem_post

[0028] S2.2.4, deinitialization queue;

[0029] S2.3. Multiple processes cannot affect each other, and channels cannot affect each other. Therefore, referring to the channel design logic of the SDK, each stream acquisition process in this method has NR_MAX_FS_CHNS channels.

[0030] S3. Main process fetches stream

[0031] First, you need to turn on the switch for taking the stream. Since there can be multiple channels for taking the stream, each channel needs a unique switch. Here, the switch is corresponding to chn+1 (number of channels + 1). When turning off the channel switch, just set the switch of the corresponding channel to 0. This ensures a non-equal design. When you want to take the stream continuously, you only need to put this closing operation in the interface of releasing the frame.

[0032] S4. Streaming process Streaming

[0033] S4.1, first of all, the virtual address space between processes is completely independent. To realize inter-process communication, it is necessary to call the system function provided by Linux, which is implemented by shared memory here; the initialization of the shared memory is placed in the SDK process. Since the virtual address space between processes is different, but the physical address space is consistent, the reference and dereference of the frame can be operated through the physical address; for the SDK main process, it is necessary to separately listen to the frame release signal of the stream acquisition process, and after listening, start to release the frame information; the main process creates a listening thread for each channel of each process, so that when the stream acquisition process releases the frame, the main process can listen and make corresponding processing;

[0034] S4.2, when the stream fetching process calls the stream fetching interface function for the first time, it needs to link the shared memory, find an idle process group, and then enable the process group element;

[0035] By default, MAXPIDNUM-1 processes are supported for stream acquisition. Considering the performance, it is limited to 6 processes. Since only two interfaces are added, releasing the stream acquisition interface cannot complete the operation of disconnecting the shared memory. Here, we check whether the processes in the process group have ended by finding the process number currently running in the system. If so, the process group can be used.

[0036] S4.3, after the stream acquisition process is initialized by calling the interface, it must first determine whether the corresponding channel of the SDK has been initialized. Only when the channel in the SDK process has been initialized, the stream acquisition process can acquire the stream;

[0037] S4.4, flow acquisition process: first turn on the flow acquisition channel switch, the switch value is chn+1; then get the flow from the queue. Since each process has its own virtual address space, and the physical address uses rmem, after obtaining the physical information, it is converted into a virtual address in each process, and then the data is processed through the physical address or virtual address;

[0038] S4.5, after fetching the stream, the fetching process needs to close the frame. Since the fetching process will mark the frame in the main process, there will be a reference to the pointer in the structure. However, since the fetching process has a separate process address space, it cannot directly operate the corresponding pointer. Therefore, when the frame needs to be released here, the listening thread in the SDK needs to be notified to handle it.

[0039] When calling the release interface of the flow acquisition process, a signal of the corresponding channel of the corresponding process is sent, and then the interface returns after waiting for the release to be completed;

[0040] S5, self-test

[0041] Process testing for the SDK process using the interface and the stream fetching process using the interface, as well as the SDK process and the stream fetching process using the interface at the same time.

[0042] In step S1, the inter-process stream acquisition is assumed to be based on Ingenic self-developed chips, and the int IMP_FrameSource_GetFrameEx (int chnNum, IMPFrameInfo**frame) function is used to obtain the stream.

[0043] int IMP_FrameSource_ReleaseFrameEx(int ​​chnNum,IMPFrameInfo*pframe) interface releases stream;

[0044] The NR_MAX_FS_CHNS macro indicates the maximum number of channels allowed on the Ingenic T series chip SDK. Different T series chips have different numbers of channels corresponding to this macro based on performance considerations. Assuming the T40 model chip platform, the macro value is 7.

[0045] In the step S1, the maximum number of frames requested is nrVBs. This parameter is the number of virtual bufs. The default value in sample is 2. If the memory allows, it can be any integer greater than or equal to 1. However, based on chip performance considerations, the current process limits the maximum number of frames to 3 without releasing frames.

[0046] In step S1, the queue is used to manage these structures. When the process calls IMP_FrameSource_GetFrameEx to get a frame from the main process, the system puts the frame into the designed queue. When the interface returns successfully, the frame is taken out of the queue and used by the process.

[0047] In step S2.1, the stream acquisition process only needs to call IMP_FrameSource_GetFrameEx to obtain the frame of the desired channel. After the main process calls this interface, the SDK will prohibit other streams from being put into the queue again until a stream acquisition process calls the IMP_FrameSource_GetFrameEx interface again.

[0048] The non-equal operation in step S3 is as follows:

[0049] When putting a frame in the queue, the frame reference count needs to be increased by 1. Only when the frame reference count is 0 will the frame information be released. The purpose of doing this is to prevent multiple modules from releasing frames at the same time, causing anomalies;

[0050] When the channel is closed, the stream interface is notified immediately to proceed to the next step. The reason for this is that when the stream interface is run twice, the second stream interface is always blocked. Therefore, it is necessary to ensure that the stream interface can continue to run after the switch is completed. Here, sem_post is used for notification processing; when releasing, just dereference the corresponding frame;

[0051] In the unbound mode, the stream acquisition function can be called to obtain the virtual address and physical address of the video frame for the algorithm to perform intelligent video processing; the bound mode refers to the method of using the IMP_System_Bind interface to obtain the stream from the system, and the unbound mode refers to the method of obtaining the stream without using the interface.

[0052] In step S4.2, the process number currently running in the system is searched to determine whether the process in the process group has ended. If so, the process group can be used. The specific method is as follows: there are files of the current process under the system / proc / path, and the size of each number is actually the pid number of the process. When the process is running, a folder with the pid name of the process will be generated under the folder. When the process ends, the corresponding folder will be deleted.

[0053] In step S4.4, converting the physical address to the virtual address includes:

[0054] The stream-taking process can obtain the starting address of rmem by reading the / proc / cmdline file. rmem is a continuous physical address. The process virtual address is obtained through the mmap function. There is a linear mapping relationship between the two. The frame uses the physical address of rmem. When the stream-taking process obtains the frame, it can find the virtual address of the frame corresponding to the process through mmap. The stream-taking process can then operate the virtual address for use.

[0055] The step S5 further includes: performing the following tests on the flow acquisition interface:

[0056] (1) Only the main process calls, which is actually thread processing. When a frame comes through the channel, the conditional lock is used for notification;

[0057] (2) Only the stream fetching process calls, and according to which stream is fetched, the frame information is assigned from the SDK stream fetching thread to the shared memory, and vbmlock protection is called at the same time. The thread in the SDK process does not call the vbmunlock frame until the release interface is called;

[0058] (3) There are both main process calls and stream fetching process calls, further including:

[0059] E. If the two processes have the same stream access channel, then: the main process must take the if main branch and fetch the frame from the shared chn through selchntorcvfrm;

[0060] F. If the two processes have different stream acquisition channels, then: First, no matter which channel is used to acquire the stream, it is necessary to ensure that the SDK has initialized the channel; the main process acquires the stream from the channel; Assumption: multiple channels have been initialized, and the main process getex stream acquisition interface can definitely obtain frame information; other processes will not enter the branch where the SDK saves mainfrmchn when acquiring the stream;

[0061] G. If the main process has the same flow channel as one flow process, but different from other flow processes, then the logic of A+B is the logic of C;

[0062] H. There is a problem with the above three points. When there are multiple stream acquisition processes in different channels, there will be competition for the variable selchntorcvfrm. Therefore, it should be possible to rewrite the parameters of only one channel in one channel, otherwise competition is bound to occur. Then:

[0063] Change the notification to use an array to control it. In the stream fetching thread, it is judged according to gpshmfrm->selchntorcvfrm[frm->poolindex]==1. If the condition is true, it is considered that the frame needs to be shared.

[0064] (4) Get debug information, semaphore count, and debug print.

[0065] Therefore, the advantages of this application are: solving the access of different platforms, reasonably using shared memory, queues, synchronization and mutual exclusion, memory mapping and other technologies, and effectively solving the needs of more complex scenario applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0067] Figure 1 It is a schematic diagram of the multi-process stream data structure of this application.

[0068] Figure 2 It is a schematic diagram of the contents in step S3 of this application.

[0069] Figure 3 It is a schematic diagram of the contents in step S5 of this application.

[0070] Figure 4 It is a flow chart of the present application method. DETAILED DESCRIPTION

[0071] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0072] like Figure 4 As shown, the present application proposes a method for solving multi-process flow acquisition at the application layer, which is divided into the following steps, and the main implementation steps are as follows:

[0073] Step S1. Inter-process communication

[0074] Because it is inter-process communication, in order to ensure maximum efficiency, the fastest shared memory communication method is adopted; because shared memory technology allows multiple processes to access the same memory area without copying data, compared with other communication methods such as pipes and message queues, it saves the overhead caused by data copying;

[0075] Get streams between processes. Based on Ingenic self-developed chips, use (int IMP_FrameSource_GetFrameEx(intchnNum,IMPFrameInfo**frame) function to get streams, int IMP_FrameSource_ReleaseFrameEx(int ​​chnNum,IMPFrameInfo*pframe) interface to release streams. Multiple processes need to be distinguished. The SDK distinguishes them by pid number. Each process corresponds to NR_MAX_FS_CHNS channels of streams. This macro indicates the maximum number of channels allowed on the Ingenic T series chip SDK. Different T series chips have different numbers of channels corresponding to this macro based on performance considerations. For example, the value of this macro is 7 for the Ingenic T40 chip platform.

[0076] In order to minimize memory consumption and ensure the continuity of frames and the stability of the project, the structure shared between multiple processes is mainly based on the frame encapsulated by the SDK. Multiple processes need to compete to obtain the right to use the frame. After the process obtains the frame, it can be used by mapping it in the space of each process. Although shared memory is very efficient, multiple processes accessing the same area of ​​shared memory at the same time will cause competition. If synchronization measures are not taken, it will inevitably cause abnormalities in the frame data obtained between each process. In order to solve the competition problem, a synchronization mechanism needs to be introduced. Commonly used synchronization mechanisms in the application layer are semaphores and mutexes. For this multi-process flow acquisition method, both methods are used. Semaphores are mainly used for synchronization and counting, while mutexes are more used for synchronization.

[0077] In order to reduce the problem of frame loss, the SDK applies for a maximum of nrVBs frames. This parameter is the number of virtual bufs. The default value in the sample is 2. If the memory allows, it can be any integer greater than or equal to 1, which is used for rotation in the system. In order to ensure that a maximum of nrVBs frames can be obtained synchronously in multi-process streaming, this function is also adapted. However, based on chip performance considerations, the current process is limited to a maximum of 3 frames without releasing frames.

[0078] In order to make the project run efficiently and correctly, that is, to more conveniently manage shared frame, semaphore, mutex and other variable information, this method puts these variable information into a gpshmframe structure and uses queues to manage its contents; when the process calls IMP_FrameSource_GetFrameEx to get the frame from the main process, the system puts the frame into the designed queue, and when the interface returns successfully, the frame is taken out of the queue and given to the process to use.

[0079] Step S2. Queue process design

[0080] S2.1. In the early stage of process design, there are two ways to handle the flow. One is "dead waiting", that is, when the main process sends the flow into the queue, the main process can continue to send the flow into the queue until the flow-taking process has taken it, processed it and released it; the other is "non-waiting", that is, after the main process sends the flow into the queue, it closes the queue channel and continues to process its own logic. The flow-taking process can also process the flow separately without interfering with each other. Obviously, the "non-waiting" design here has higher performance, so it needs to be made into a "non-waiting" operation. For the flow-taking process, you only need to call IMP_FrameSource_GetFrameEx to get the frame of the desired channel. After the main process calls this interface, the SDK will prohibit other flows from being put into the queue again until a flow-taking process calls the IMP_FrameSource_GetFrameEx interface again;

[0081] S2.2. In the above-mentioned process of entering and exiting the queue, synchronization still needs to be ensured, as shown below: The process of using the queue includes:

[0082] S2.2.1, Initialize the queue

[0083] S2.2.2, Enqueue Operation:

[0084] Sem_wait

[0085] Lock

[0086] Add data to the queue

[0087] Notify the dequeue operation to fetch the stream (cond / sem_post)

[0088] Unlock

[0089] S2.2.3, Dequeue operation:

[0090] Lock

[0091] Wait until the queue operation notifies the user to fetch the stream

[0092] Retrieve data from the queue

[0093] Unlock

[0094] Sem_post

[0095] S2.2.4, Deinitialize queue.

[0096] S2.3. Multiple processes cannot affect each other, and channels cannot affect each other. Therefore, referring to the channel design logic of the SDK, it is generally as follows Figure 1In this method, each stream acquisition process has NR_MAX_FS_CHNS (the maximum number of channels on the Ingenic platform) channels.

[0097] Step S3. Main process gets the stream

[0098] S3. Main process fetches stream

[0099] First, you need to turn on the switch for taking the flow. Because there can be multiple channels for taking the flow, each channel needs a unique switch. Here, the switch is corresponding to chn+1 (number of channels + 1); when turning off the channel switch, just set the switch of the corresponding channel to 0; Figure 2 As shown, this ensures the "non-equal" design. When you want to obtain the stream continuously, you only need to put the closing operation into the interface of the release frame.

[0100] The non-equal operations are as follows:

[0101] When putting a frame in the queue, the frame reference count needs to be increased by 1. Only when the frame reference count is 0 will the frame information be released. The purpose of doing this is to prevent multiple modules from releasing frames at the same time, causing anomalies;

[0102] When the channel is closed, the stream interface is immediately notified to proceed to the next step. The reason for doing this is that when the stream interface runs twice, the second stream interface is always in a blocked state. Therefore, it is necessary to ensure that the stream interface can continue to run after the switch is completed. Here, sem_post is used for notification processing; when releasing, just dereference the corresponding frame.

[0103] In the unbound mode, you can call the stream acquisition function to obtain the virtual address and physical address of the video frame for the algorithm to perform intelligent video processing. The bound mode refers to the method of using the IMP_System_Bind interface to obtain the stream from the system, and the unbound mode refers to the method of obtaining the stream without using the interface.

[0104] Step S4. Stream acquisition process

[0105] For the flow acquisition process, the process is slightly more complicated:

[0106] S4.1, first of all, the virtual address space between processes is completely independent. To achieve inter-process communication, it is necessary to call the system function provided by Linux, which is implemented here by shared memory; the initialization of the shared memory is placed in the SDK process. Since the virtual address space between processes is different, but the physical address space is consistent, the reference and dereference of the frame can be operated through the physical address. For the SDK main process, it is necessary to separately listen to the frame release signal of the stream-taking process, and after listening, start to release the frame information; the main process creates a listening thread for each channel of each process, so that when the stream-taking process releases the frame, the main process can listen and make corresponding processing; here, the process using the IMP_System_Init interface is called the main process, and the process that does not use IMP_System_Init initialization but directly uses IMP_FrameSource_GetFrameEx is called the stream-taking process;

[0107] S4.2, when the stream fetching process calls the stream fetching interface function for the first time, it needs to link the shared memory, find an idle process group, and then enable the process group element;

[0108] By default, MAXPIDNUM-1 processes are supported for stream acquisition. Considering the performance, it is limited to 6 processes. Since only two interfaces are added, releasing the stream acquisition interface cannot complete the operation of disconnecting the shared memory. Here, we check whether the processes in the process group have ended by finding the process number currently running in the system. If so, the process group can be used.

[0109] Here, we can determine whether the process in the process group has ended by finding the process number currently running in the system. If it has ended, the process group can be used. The specific method is as follows: There are files of the current process in the system / proc / path. The size of each number is actually the pid number of the process. When the process is running, a folder with the pid name of the process will be generated in this folder. When the process ends, the corresponding folder will be deleted.

[0110] S4.3, after the stream acquisition process is initialized by calling the interface, it must first determine whether the corresponding channel of the SDK has been initialized. Only when the channel in the SDK process has been initialized, the stream acquisition process can acquire the stream;

[0111] S4.4, flow acquisition process: first turn on the flow acquisition channel switch, the switch value is chn+1; then get the flow from the queue. Since each process has its own virtual address space, and the physical address uses rmem, after obtaining the physical information, it is converted into a virtual address in each process, and then the data is processed through the physical address or virtual address;

[0112] The converting of the physical address to the virtual address includes:

[0113] The stream-taking process can obtain the starting address of rmem by reading the / proc / cmdline file. rmem is a continuous physical address. Then the process virtual address is obtained through the mmap function. There is a linear mapping relationship between the two. The frame uses the physical address of rmem. When the stream-taking process obtains the frame, it can find the virtual address of the frame corresponding to the process through mmap. The stream-taking process can then operate the virtual address for use.

[0114] S4.5, after fetching the stream, the fetching process needs to close the frame. Since the fetching process will mark the frame in the main process, there will be a reference to the pointer in the structure. However, since the fetching process has a separate process address space, it cannot directly operate the corresponding pointer. Therefore, when the frame needs to be released here, the listening thread in the SDK needs to be notified to handle it.

[0115] When calling the release interface of the stream acquisition process, a signal of the corresponding channel of the corresponding process is sent, and then the interface returns after waiting for the release to be completed.

[0116] Step S5. Self-test

[0117] Test the SDK process using the interface and the stream fetching process using the interface, as well as the SDK process and the stream fetching process using the interface at the same time. Figure 3 As shown in the figure, the following tests are performed on the stream acquisition interface: (1) Only the main process calls, which is actually thread processing. When a frame comes through the channel, a conditional lock is used for notification;

[0118] (2) Only the stream fetching process calls, and according to which stream is fetched, the frame information is assigned from the SDK stream fetching thread to the shared memory, and vbmlock protection is called at the same time. The thread in the SDK process does not call the vbmunlock frame until the release interface is called;

[0119] (3) There are both main process calls and stream fetching process calls, further including: I. If the two processes have the same stream fetching channel, then: the main process fetching stream must go through the if main branch and fetch the frame from the shared chn through selchntorcvfrm;

[0120] J. If the two processes have different stream acquisition channels, then: First, no matter which channel is used to acquire the stream, it is necessary to ensure that the SDK has initialized the channel; the main process acquires the stream from the channel; Assumption: multiple channels have been initialized, and the main process getex stream acquisition interface can definitely obtain frame information; other processes will not enter the branch where the SDK saves mainfrmchn when acquiring the stream;

[0121] K. If the main process has the same flow channel as one flow process, but different from other flow processes, then the logic of A+B is the logic of C;

[0122] L. There is a problem with the above three points. When there are multiple stream acquisition processes and different channels, there will be competition for the variable selchntorcvfrm. Therefore, it should be possible to rewrite the parameters of only one channel in one channel, otherwise competition is bound to occur. Then:

[0123] Change the notification to use an array to control it. In the stream fetching thread, it is judged according to gpshmfrm->selchntorcvfrm[frm->poolindex]==1. If the condition is true, it is considered that the frame needs to be shared.

[0124] (4) Get debug information, semaphore count, and debug print.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for solving multi-process flow acquisition at the application layer, characterized in that: The method comprises the following steps: S1. Inter-process communication: Because it is inter-process communication, the fastest shared memory communication method is used to ensure maximum efficiency; To obtain streams between processes, multiple processes need to be distinguished. The SDK uses the pid number to distinguish and process. Each process corresponds to the streams of NR_MAX_FS_CHNS channels. This macro indicates the maximum number of channels allowed on the chip SDK. Different chips have different performances, and the number of channels corresponding to this macro is also different. The structure shared between multiple processes in this method is based on the frame encapsulated by the SDK. Multiple processes need to compete to obtain the right to use the frame. In order to solve the competition problem, a synchronization mechanism is introduced. For this method of using multiple processes to obtain streams, the synchronization mechanisms used in the application layer include semaphores and mutexes. Semaphores are used for synchronization and counting, and mutexes are used for synchronization. In order to reduce the problem of frame loss, the SDK applies for a maximum of nrVBs frames for rotation in the system; in order to ensure synchronization, the multi-process stream fetching can fetch a maximum of nrVBs frames, and this function is also adapted; in order to more conveniently manage the shared frame, semaphore, mutex and other variable information, this method puts these variable information into a gpshmframe structure and uses a queue to manage the content; S2. Queue process design, including: S2.

1. In order to prevent the main process from being affected when the stream fetching process fetches the stream, a non-equal operation is required. After fetching a frame, the main process puts the stream into the queue and closes it until the fetching interface is called again. S2.

2. Since it is a queue operation between multiple processes, the SDK enters the queue and the stream retrieval process exits the queue, so synchronization needs to be ensured. The ways to ensure synchronization between process communications include synchronization locks and semaphores. The process of using queues includes: S2.2.1, Initialize the queue S2.2.2, Enqueue Operation: Sem_wait Lock Add data to the queue Notify the dequeue operation to fetch the stream (cond / sem_post) Unlock S2.2.3, Dequeue operation: Lock Wait until the queue operation notifies the stream to be taken Get data from the queue Unlock Sem_post S2.2.4, deinitialization queue; S2.

3. Multiple processes cannot affect each other, and channels cannot affect each other. Therefore, referring to the channel design logic of the SDK, each stream acquisition process in this method has NR_MAX_FS_CHNS channels. S3. Main process fetches stream First, you need to turn on the switch for taking the stream. Because there can be multiple channels for taking the stream, each channel needs a unique switch. Here, the switch is corresponding to chn+1, that is, the number of channels + 1. When turning off the channel switch, just set the switch of the corresponding channel to 0. This ensures a non-equal design. When you want to take the stream continuously, you only need to put this closing operation in the interface of releasing the frame. S4. Streaming process Streaming S4.1, first of all, the virtual address space between processes is completely independent. To realize inter-process communication, it is necessary to call the system function provided by Linux, which is implemented by shared memory here; the initialization of the shared memory is placed in the SDK process. Since the virtual address space between processes is different, but the physical address space is consistent, the reference and dereference of the frame can be operated through the physical address; for the SDK main process, it is necessary to separately listen to the frame release signal of the stream acquisition process, and after listening, start to release the frame information; the main process creates a listening thread for each channel of each process, so that when the stream acquisition process releases the frame, the main process can listen and make corresponding processing; S4.2, when the stream fetching process calls the stream fetching interface function for the first time, it needs to link the shared memory, find an idle process group, and then enable the process group element; By default, MAXPIDNUM-1 processes are supported for stream acquisition. Considering the performance, it is limited to 6 processes. Since only two interfaces are added, releasing the stream acquisition interface cannot complete the operation of disconnecting the shared memory. Here, we check whether the processes in the process group have ended by finding the process number currently running in the system. If so, the process group can be used. S4.3, after the stream acquisition process is initialized by calling the interface, it must first determine whether the corresponding channel of the SDK has been initialized. Only when the channel in the SDK process has been initialized, the stream acquisition process can acquire the stream; S4.4, flow acquisition process: first turn on the flow acquisition channel switch, the switch value is chn+1; then get the flow from the queue. Since each process has its own virtual address space, and the physical address uses rmem, after obtaining the physical information, it is converted into a virtual address in each process, and then the data is processed through the physical address or virtual address; S4.5, after fetching the stream, the fetching process needs to close the frame. Since the fetching process will mark the frame in the main process, there will be a reference to the pointer in the structure. However, since the fetching process has a separate process address space, it cannot directly operate the corresponding pointer. Therefore, when the frame needs to be released here, the listening thread in the SDK needs to be notified to handle it. When calling the release interface of the flow acquisition process, a signal of the corresponding channel of the corresponding process is sent, and then the interface returns after waiting for the release to be completed; S5, self-test Process testing for the SDK process using the interface and the stream fetching process using the interface, as well as the SDK process and the stream fetching process using the interface at the same time.

2. According to claim 1, a method for solving multi-process flow acquisition at the application layer is characterized in that: In step S1, the inter-process stream acquisition is assumed to be based on Ingenic self-developed chips, and the int IMP_FrameSource_GetFrameEx (intchnNum, IMPFrameInfo**frame) function is used to obtain the stream. int IMP_FrameSource_ReleaseFrameEx(int ​​chnNum,IMPFrameInfo*pframe) interface releases stream; The NR_MAX_FS_CHNS macro indicates the maximum number of channels allowed on the Ingenic T series chip SDK. Different T series chips have different numbers of channels corresponding to this macro based on performance considerations. Assuming the T40 model chip platform, the macro value is 7.

3. The method for solving multi-process flow acquisition at the application layer according to claim 2 is characterized in that: In the step S1, the maximum number of frames requested is nrVBs. This parameter is the number of virtual bufs. The default value in sample is 2. If the memory allows, it can be any integer greater than or equal to 1. However, based on chip performance considerations, the current process limits the maximum number of frames to 3 without releasing frames.

4. The method for solving multi-process flow acquisition at the application layer according to claim 3 is characterized in that: In step S1, the queue is used to manage these structures. When the process calls IMP_FrameSource_GetFrameEx to get a frame from the main process, the system puts the frame into the designed queue. When the interface returns successfully, the frame is taken out of the queue and used by the process.

5. The method for solving multi-process flow acquisition at the application layer according to claim 4 is characterized in that: In step S2.1, the stream acquisition process only needs to call IMP_FrameSource_GetFrameEx to obtain the frame of the desired channel. After the main process calls this interface, the SDK will prohibit other streams from being put into the queue again until a stream acquisition process calls the IMP_FrameSource_GetFrameEx interface again.

6. The method for solving multi-process flow acquisition at the application layer according to claim 1 is characterized in that: The non-equal operation in step S3 is as follows: When putting a frame in the queue, the frame reference count needs to be increased by 1. Only when the frame reference count is 0 will the frame information be released. The purpose of doing this is to prevent multiple modules from releasing frames at the same time, causing anomalies; When the channel is closed, the stream interface is notified immediately to proceed to the next step. The reason for this is that when the stream interface is run twice, the second stream interface is always blocked. Therefore, it is necessary to ensure that the stream interface can continue to run after the switch is completed. Here, sem_post is used for notification processing; when releasing, just dereference the corresponding frame; In the unbound mode, the stream acquisition function can be called to obtain the virtual address and physical address of the video frame for the algorithm to perform intelligent video processing; the bound mode refers to the method of using the IMP_System_Bind interface to obtain the stream from the system, and the unbound mode refers to the method of obtaining the stream without using the interface.

7. The method for solving multi-process flow acquisition at the application layer according to claim 1, characterized in that: In step S4.2, the process number currently running in the system is searched to determine whether the process in the process group has ended. If so, the process group can be used. The specific method is as follows: there are files of the current process under the system / proc / path, and the size of each number is actually the pid number of the process. When the process is running, a folder with the pid name of the process will be generated under the folder. When the process ends, the corresponding folder will be deleted.

8. The method for solving multi-process flow acquisition at the application layer according to claim 7 is characterized in that: In step S4.4, converting the physical address to the virtual address includes: The stream-taking process can obtain the starting address of rmem by reading the / proc / cmdline file. rmem is a continuous physical address. The process virtual address is obtained through the mmap function. There is a linear mapping relationship between the two. The frame uses the physical address of rmem. When the stream-taking process obtains the frame, it can find the virtual address of the frame corresponding to the process through mmap. The stream-taking process can then operate the virtual address for use.

9. The method for solving multi-process flow acquisition at the application layer according to claim 1, characterized in that: The step S5 further includes: performing the following tests on the flow acquisition interface: (1) Only the main process calls, which is actually thread processing. When a frame comes through the channel, the conditional lock is used for notification; (2) Only the stream fetching process calls, and according to which stream is fetched, the frame information is assigned from the SDK stream fetching thread to the shared memory, and vbmlock protection is called at the same time. The thread in the SDK process does not call the vbmunlock frame until the release interface is called; (3) There are both main process calls and stream fetching process calls, further including: A. If the two processes have the same stream access channel, then: the main process must take the if main branch and use selchntorcvfrm to get the frame from the shared chn; B. If the two processes have different stream acquisition channels, then: First, no matter which channel is used to acquire the stream, it is necessary to ensure that the SDK has initialized the channel; the main process acquires the stream from the channel; Assumption: multiple channels have been initialized, and the main process getex stream acquisition interface can definitely obtain frame information; other processes will not enter the branch where the SDK saves mainfrmchn when acquiring the stream; C. If the main process has the same flow channel as one flow process, but different from other flow processes, then the logic of A+B is the logic of C; D. There is a problem with the above three points. When there are multiple stream acquisition processes in different channels, there will be competition for the variable selchntorcvfrm. Therefore, it should be possible to rewrite the parameters of only one channel in one channel, otherwise competition is bound to occur. Then: Change the notification to use an array to control it. In the stream fetching thread, it is judged according to gpshmfrm->selchntorcvfrm[frm->poolindex]==1. If the condition is true, it is considered that the frame needs to be shared. (4) Get debug information, semaphore count, and debug print.

Images