Multi-channel video processing method and apparatus, and device, tool, medium and product

Abstract

Disclosed in the present disclosure are a multi-channel video processing method and apparatus, and a device, a tool, a medium and a product. The method comprises: acquiring videos of a plurality of channels, wherein the plurality of channels comprise at least one main channel and a plurality of auxiliary channels, and videos of the main channel and videos of the auxiliary channels are respectively collected using a first camera device and second camera device of a predetermined scenario; processing image frames of the videos of the plurality of channels using processing resources, so as to obtain processing results of the channels, wherein the image frames of the videos of the plurality of auxiliary channels are processed in a time division processing mode; and integrating the processing results of the channels, so as to obtain a target processing result. By means of the solution, the computation amount of multi-channel video processing can be reduced.

Description

Multi-channel video processing methods, devices, equipment, tools, media and products

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on the Chinese patent application with application number 202311764028.8, application date December 20, 2023, and application name “Multi-channel video processing methods, devices, equipment, tools, media and products”, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into the present disclosure in its entirety. Technical Field

[0003] The present disclosure relates to the field of image processing technology, and in particular to a multi-channel video processing method, device, equipment, tool, medium and product. Background Art

[0004] With the development of computer technology, image processing technology has been widely used in people's daily lives, and the demand for image or video processing has also increased.

[0005] Currently, some devices support image processing functions such as image detection and image recognition. When processing multiple image frames or multiple channels of video from different acquisition devices, this typically consumes significant processing resources. However, due to the limited processing resources of these devices and the high computational complexity of image processing, excessive processing resources can compromise image processing effectiveness and efficiency.

[0006] Summary of the Invention

[0007] The main technical problem solved by the present disclosure is to provide a multi-channel video processing method, device, equipment, tool, medium and product, which can reduce the computational complexity of multi-channel video image processing.

[0008] In order to solve the above problems, the first aspect of an embodiment of the present disclosure provides a multi-channel video processing method, which includes: obtaining videos of multiple channels, the multiple channels including at least one main channel and multiple auxiliary channels, the videos of the main channel and the videos of the auxiliary channels are respectively captured by a first camera device and a second camera device of a predetermined scene; using processing resources to process image frames of the videos of the multiple channels to obtain processing results of each channel, wherein the processing of the image frames of the multiple auxiliary channel videos adopts a time-division processing method; and integrating the processing results of each channel to obtain a target processing result.

[0009] The above scheme obtains the target videos of multiple channels, uses processing resources to process the image frames of the video of the main channel and adopts time-division processing to process the image frames of the videos of multiple auxiliary channels to obtain the processing results of each channel. The videos of multiple channels can be processed separately at multiple moments in time, and then the processing results of the main channel and each auxiliary channel are combined to obtain the target processing results. Compared with processing the target videos of multiple channels at the same time, the amount of image processing calculations can be reduced, thereby reducing the consumption of processing resources and improving processing efficiency.

[0010] In order to solve the above problems, the second aspect of an embodiment of the present disclosure provides a multi-channel video processing device, which includes: an acquisition unit, a processing unit and an integration unit, the acquisition unit is used to acquire videos of multiple channels, the multiple channels include at least one main channel and multiple auxiliary channels, the videos of the main channel and the videos of the auxiliary channels are respectively captured by a first camera device and a second camera device of a predetermined scene; the processing unit is used to use processing resources to process image frames of the videos of the multiple channels to obtain processing results of each channel, wherein the processing of the image frames of the multiple auxiliary channel videos adopts a time-division processing method; the integration unit is used to integrate the processing results of each channel to obtain a target processing result.

[0011] A third aspect of an embodiment of the present disclosure provides a computer device, which includes a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement any step of the above-mentioned multi-channel video processing method.

[0012] A fourth aspect of an embodiment of the present disclosure provides a vehicle, which includes multiple camera devices and a processing module, wherein the multiple camera devices include the first camera device and the second camera device, and the processing module is used to provide processing resources and to run program data to perform any step of the above-mentioned multi-channel video processing method.

[0013] In order to solve the above problems, the fifth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, which stores program data that can be executed by a processor, and the program data is used to implement any step of the above multi-channel video processing method.

[0014] A sixth aspect of the embodiments of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above-mentioned multi-channel video processing method.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0017] FIG1 is a flow chart of an embodiment of a multi-channel video processing method disclosed herein;

[0018] FIG2 is a flow chart of an embodiment of step S12 of the present disclosure;

[0019] FIG3 is a schematic diagram of preset processing frame rates for each channel according to an embodiment of the present disclosure;

[0020] FIG4 is a flow chart of an embodiment of step S13 of the present disclosure;

[0021] FIG5 is a schematic diagram of preset processing frame rates for each channel according to another embodiment of the present disclosure;

[0022] FIG6 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0023] FIG7 is a schematic structural diagram of an embodiment of a vehicle disclosed herein;

[0024] FIG8 is a schematic structural diagram of an embodiment of a computer device disclosed herein;

[0025] FIG9 is a schematic structural diagram of an embodiment of a computer-readable storage medium disclosed herein. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0027] The terms "first" and "second" in this disclosure are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0028] References to "embodiments" in this disclosure mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the disclosure. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] The present disclosure provides the following embodiments, and each embodiment is described in detail below.

[0031] The multi-channel video processing method of the embodiments of the present disclosure may be executed by an image processing device, which may be provided in any computer device, server, vehicle, or other processing device capable of executing the embodiments of the present disclosure. The computer device may be a user device, mobile device, user terminal, vehicle-mounted device, or the like, and the vehicle may be a vehicle. In some possible implementations, the multi-channel video processing method may be implemented by a processor calling program data stored in a memory.

[0032] Please refer to Figure 1, which is a flow chart of an embodiment of a multi-channel video processing method disclosed herein. The method may include the following steps:

[0033] S11: Acquire videos of multiple channels.

[0034] Here, the channel represents the source of the video, and each channel corresponds to one channel of video data. In an actual scenario, multiple cameras can be set up, and the multiple cameras respectively capture video signals, and the image frame sequence continuously captured by each camera forms the video of a corresponding channel. In this embodiment, the multiple channels include at least one main channel and multiple auxiliary channels, and the video of the main channel is captured by the first camera of the predetermined scene, and the video of the auxiliary channel is captured by the second camera of the above-mentioned predetermined scene. The first camera and the second camera can be cameras with different viewing angles in the same predetermined scene. In an actual scenario, the first camera and the second camera can be set up at a fixed position in the preset scene, or loaded on a movable device in the preset scene. The movable device can be, for example, a means of transportation (such as a vehicle), a mobile robot, etc.

[0035] The first and second cameras are used to capture the scene in which the target device is located, thereby acquiring video corresponding to the channels of each camera. This method enables the acquisition of video from multiple channels. The camera can be a video camera, a still camera, a surveillance camera, or other devices with a recording function. The type of camera can be selected based on the specific application scenario, and this disclosure does not impose any particular restrictions on this.

[0036] In some embodiments, videos from different channels are captured using cameras with different fields of view, where the fields of view of the cameras corresponding to the different channels at least partially overlap. Because videos from different channels are captured using cameras with different fields of view, images from different fields of view can be processed and the image processing results from these different fields of view can be combined to obtain image processing results for a wider range of fields of view. In some embodiments, when multiple cameras corresponding to different channels have at least partially overlapping fields of view, more accurate global video processing results for a predetermined scene can be obtained.

[0037] In some embodiments, the field of view of the first camera device and the field of view of each second camera device at least partially overlap. The first camera device may be used to capture an image of the central field of view of a predetermined scene, and the second camera device may be used to capture images of the side fields of view of the predetermined scene. The fields of view of each second camera device may not overlap.

[0038] Taking the driving scene as an example, the first camera device corresponding to the main channel can be set at the central position of the vehicle, such as the center of the front windshield of the vehicle as high as possible, the front top position of the vehicle, etc., and its shooting field of view is the field of view in the direction of vehicle travel, such as the field of view in front or behind the vehicle; the second camera device corresponding to at least one auxiliary channel can be respectively set at the side position of the vehicle, such as the left side, right side, the front (rear) of the left side, the front (rear) of the right side, etc., and its shooting field of view is the side field of view of the vehicle. It can be understood by those skilled in the art that the installation position of each camera device can be set according to the specific application scenario or requirements, and the present disclosure does not limit this.

[0039] Exemplarily, in the case of multi-channel video processing for target detection or recognition, the object to be detected or recognized in the image captured by the second camera device is a portion of the object to be detected or recognized in the image captured by the first camera device. Optionally, the object to be recognized in the image captured by the second camera device can be recognized through local information and can be static. Taking a driving scene as an example, the object to be recognized by the second camera device can be any one of free space, lane lines, zebra areas, road markings, etc.

[0040] In the above scheme, the first camera device can capture images of a wider central field of view, and at least one second camera device can capture images of the side field of view of the target device, which can supplement the blind spot of the field of view of the main camera device, so that the image processing results of different shooting fields of view can be combined to obtain image processing results of a more comprehensive field of view within the predetermined scene.

[0041] In some application scenarios, such as a 360° panoramic monitoring system, the first camera device and the second camera device can be distributed in multiple different shooting angles of a preset scene, such as front view, rear view, left view, right view, etc. The first camera device and several second camera devices can be determined from multiple camera devices of the preset scene. For example, a camera device with a larger shooting field of view or a more important shooting field of view is used as the first camera device, and a camera device with a smaller shooting field of view or a relatively less important shooting field of view is used as the second camera device, etc. The first camera device and several second camera devices can be determined based on the specific needs of the user or the application scenario, etc., and the present disclosure does not impose any restrictions on this.

[0042] S12: Processing the image frames of the videos of the multiple channels using processing resources to obtain processing results of each channel, wherein the image frames of the videos of the multiple auxiliary channels are processed using a time division processing method.

[0043] Processing resources are resources that can be used for image processing, such as the computational processing resources of a processor. These resources can be used to process the image frames of multiple auxiliary channels using a time-sharing approach to obtain the time-sharing processing results for each auxiliary channel during its corresponding processing time period. Simultaneously, these processing resources are used to synchronously process the image frames of the main channel's video.

[0044] Time-division processing involves switching processing resources to process the image frames of each channel's video at specific intervals. This reduces the amount of image processing required per period and sequentially processes the image frames of each auxiliary channel, effectively improving resource processing efficiency.

[0045] The aforementioned method for processing the videos of the main channel and the auxiliary channels using processing resources may be, for example, to create multiple resource scheduling units, with each main channel video occupying a separate resource scheduling unit, and the videos of multiple auxiliary channels sharing a resource scheduling unit; or to divide the multiple auxiliary channels into multiple channel groups, with each channel group occupying a resource scheduling unit, and the videos of all auxiliary channels in the same channel group sharing a resource scheduling unit. Here, the resource scheduling unit may be, for example, a process or a thread.

[0046] In some embodiments, referring to FIG2 , step S12 of the above embodiment can be further expanded. Using processing resources to process image frames of videos of multiple channels to obtain processing results of each channel, this embodiment may include the following steps:

[0047] S121: At each current moment in the process of processing the videos of the multiple channels, determine the processing channel and the non-processing channel at the current moment from the multiple channels according to the preset processing frame rates of the multiple channels respectively; wherein the image frames of the videos of the non-processing channels are not processed at the current moment.

[0048] You can get the preset processing frame rate for each channel. The preset processing frame rate indicates the number of image frames processed per unit time. For example, a preset processing frame rate of 2 means that 2 frames in the video are processed per unit time.

[0049] In this embodiment, the preset processing frame rates of multiple auxiliary channels may be the same. According to the time-division processing method, the processing time periods of the videos of multiple different auxiliary channels may not overlap at all. The processing time period of the main channel and the video may overlap or not overlap with the processing time period of the videos of each auxiliary channel. In a specific implementation, a processing cycle may be set, and the processing cycle may be set, for example, according to the preset processing frame rate of the main channel. Within one processing cycle, it is necessary to complete the processing of at least one image frame in the video of the main channel, and complete the processing of one frame of image in the video of each auxiliary channel in turn. In this way, the processing moment of each channel within a processing cycle can be determined according to the preset processing frame rate of each channel mentioned above, and then, the processing channel and non-processing channel at the current moment can be determined according to the processing moment. Among them, the image frames of the video of the processing channel are processed at the current moment, the image frames of the video of the non-processing channel are not processed at the current moment, and the non-processing channel contains at least one channel.

[0050] In some embodiments, the preset processing frame rate of the primary channel is a first processing frame rate, and the preset processing frame rate of the auxiliary channel is a second processing frame rate. In some application scenarios, because the camera corresponding to the primary channel has a wider field of view than the auxiliary channel, the second processing frame rate can be set to be lower than the first processing frame rate, so that the number of image frames processed for the primary channel in the same time period is greater than the number of image frames processed for a single auxiliary channel.

[0051] For the primary channel, a determination is made based on the first processing frame rate as to whether the current moment is the processing moment of the primary channel. For example, if the processing moment of the primary channel is T1, T2, T3, T4, etc., then when the current moment is T1, a determination is made as to whether the current moment T1 is the processing moment of the primary channel. Subsequently, at subsequent moments T2, T3, T4, etc., a determination is made as to whether the current moment is the processing moment of the primary channel. If so, then, in response to the current moment being the processing moment of the primary channel, the primary channel may be designated as the primary processing channel at the current moment.

[0052] For auxiliary channels, the second processing frame rate is used to determine whether the current time is the processing time for each auxiliary channel. For example, if the processing time of an auxiliary channel is T2, T4, etc., when the current time is T1, the process determines whether the current time T1 is the processing time for the auxiliary channel. If not, no subsequent processing is performed, and the above steps are continued at the next time T2. That is, when the current time is T2, the process determines whether the current time T2 is the processing time for the auxiliary channel. If so, the auxiliary channel whose current time is the processing time can be used as the second processing channel at the current time. The second processing frame rate is lower than the first processing frame rate.

[0053] The above scheme determines the processing channel at each current moment in the processing process by using different processing frame rates for the main channel and the auxiliary channel respectively, so as to realize time-division processing of the auxiliary channel. In addition, since the second processing frame rate is lower than the first processing frame rate, the number of image processing for a single auxiliary channel can be reduced while maintaining the number of image processing for the main channel within a processing cycle, thereby reducing the amount of image processing computation.

[0054] In some embodiments, there are multiple auxiliary channels, and the multiple auxiliary channels can be divided into multiple auxiliary channel groups. During the processing period corresponding to the second processing frame rate, the auxiliary channels in the same group are processed at the same time, and the processing times of the auxiliary channels in different groups do not overlap. This approach facilitates time-sharing processing of image frames of the videos of the auxiliary channels in different groups in subsequent processes.

[0055] The first processing frame rate may be N times the second processing frame rate, where N is an integer greater than 1. Optionally, N is the number of auxiliary channel groups. Within the processing period corresponding to the second processing frame rate, the processing moment of the main channel overlaps once with the processing moment of each auxiliary channel group. For example, assuming there is a main channel and m auxiliary channel groups (m is an integer greater than 1), the processing moments of the main channel are t1, t2, t3, ..., and the processing moments of the i-th auxiliary channel group are ti, t(m+i), t(2m+i), ..., where i = 1, 2, ..., m.

[0056] The above solution divides multiple auxiliary channels into multiple groups and determines the processing cycle according to a second preset frame rate. Each auxiliary channel group has the same processing time within the same processing cycle, and the processing times of different auxiliary channel groups do not overlap, thereby achieving time-division processing of multiple auxiliary channel groups. The first processing frame rate is N times the second processing frame rate, which can achieve synchronous processing of the video of the main channel and the video of the auxiliary channels. While ensuring that the videos of at least two channels, including the main channel, are processed at the same processing time, the image frames in the videos of each auxiliary channel are processed in a polling manner throughout the entire processing process, thereby ensuring the comprehensiveness of the final integrated processing results.

[0057] As an example, please refer to Figure 3. Multiple channels include a main channel CC and multiple auxiliary channels (such as SC1 and SC2). The first processing frame rate corresponding to the main channel is twice the second processing frame rate corresponding to the auxiliary channel. For example, within a unit of time, the first processing frame rate can be set to 4 and the second processing frame rate can be set to 2. The processing time of the main channel CC is T1, T2, T3, T4, etc., the processing time of the auxiliary channel SC1 is T1, T3, etc., and the processing time of the auxiliary channel SC2 is T2, T4, etc. When the current time is T1, the main channel CC can be determined as the first processing channel and the auxiliary channel SC1 as the second processing channel; when the current time is T2, the main channel CC can be determined as the first processing channel and the auxiliary channel SC2 as the second processing channel. Similarly, the processing channel at each moment in the time-division processing method can be determined.

[0058] S122: Processing the image frames of the video of the processing channel at the current moment using processing resources.

[0059] After determining the processing channel and non-processing channel at the current moment, the processing resources can be used to process the image frames of the video of the processing channel at the current moment to obtain the processing result at the current moment. The processing result at the current moment can refer to the processing result of the image frames in the video of the processing channel at the current moment.

[0060] The above solution realizes time-sharing processing by separately determining the processing channel and the non-processing channel. Compared with the method of processing all channel videos at the same time, it reduces the number of image frames processed at the same time and reduces the amount of image processing calculations.

[0061] In some embodiments, processing resources include primary processing resources and secondary processing resources, which may correspond to different processing units. The primary processing resources and secondary processing resources may be, for example, primary and secondary threads, or primary and secondary processes, respectively. The primary processing resources may be used to perform image processing on a primary channel, while the secondary processing resources may be used to perform image processing on a secondary channel.

[0062] In some embodiments, the preset processing frame rate of each channel can be dynamically adjusted based on changes in the scene. In this process, before step S121, the preset processing frame rate at the current moment is obtained. Specifically, changes in the current scene state can be obtained, such as changes in the direction, speed, or posture of the device that sets up the first camera and the second camera in the scene. Then, when the scene state of the predetermined scene meets predetermined conditions, the preset processing frame rate corresponding to at least one auxiliary channel is adjusted based on the scene state.

[0063] Furthermore, when the predetermined scene is a driving scene, the first camera device is used to capture video of the vehicle's driving direction, and the second camera device is used to capture video of the vehicle's side. For example, the first camera device is set on the top of the vehicle and faces the front or rear of the vehicle, and multiple second cameras are set at multiple different positions on the side of the vehicle and face multiple different angles on both sides of the vehicle. The scene state includes the driving state of the vehicle. When the driving state of the vehicle is turning, the preset processing frame rate of the auxiliary channel corresponding to the second camera device for capturing video of the vehicle turning to one side is increased, and / or the preset processing frame rate of the auxiliary channel corresponding to the second camera device for capturing video of the vehicle turning away from one side is reduced.

[0064] For example, when the vehicle turns right, the processing frame rate of each auxiliary channel corresponding to the camera equipment on the side of the vehicle is adjusted so that the processing frame rate corresponding to the channel of the camera equipment on the right side is higher than the processing frame rate corresponding to the channel of the camera equipment on the left side. Therefore, when the vehicle turns, by increasing the processing frame rate of the video on the turning side, the number of processing frames of the road video on the turning side in one processing cycle is increased, and the road conditions on the turning side are more fully identified or detected, which helps to improve the safety of vehicle turning decisions in autonomous driving or assisted driving scenarios.

[0065] Returning to FIG. 1 , in step S13 , the processing results of each channel are integrated to obtain the target processing result.

[0066] The processing results of each channel can be integrated to obtain the target processing result. Specifically, the processing results of each channel can be projected into a common coordinate system, for example, into a bird's-eye view coordinate system to form a bird's-eye view, thereby obtaining the target processing result.

[0067] The above scheme obtains videos from multiple channels, uses processing resources to time-share the image frames of the videos of multiple auxiliary channels, obtains the time-shared processing results of each auxiliary channel, and simultaneously processes the image frames of the video of the main channel to obtain the processing results of the main channel. Then, the processing results of the main channel and the time-shared processing results of each auxiliary channel are combined to obtain the target processing results. Compared with processing videos of multiple channels at the same time, the number of images processed at the same time can be reduced, that is, the amount of image processing calculations can be reduced, thereby reducing the consumption of processing resources and reducing the computing load.

[0068] In some embodiments, referring to FIG4 , step S13 of the above embodiment can be further expanded. In this embodiment, the process of integrating the processing results of each channel may include the following steps:

[0069] S131: For each current moment in the process of processing the videos of the multiple channels, determine the processing result of the processing channel at the current moment from each channel, and determine the interpolation processing result of the non-processing channel at the current moment, wherein the interpolation processing result is obtained by interpolating the posture information of the camera device corresponding to the non-processing channel and the processing results of the non-processing channel at the previous moment and / or subsequent moment of the current moment.

[0070] The processing results of the processing channels are obtained by executing the above step S12, such as the processing results of the first processing channel and the processing results of the second processing channel.

[0071] For the non-processed channel at the current moment, no processing results are available at the current moment because the image frames of the video of the non-processed channel have not been processed at the current moment. In determining the interpolation processing result for the non-processed channel at the current moment, the interpolation processing result for the non-processed channel at the current moment can be obtained by interpolating the position information of the camera device corresponding to the non-processed channel and the time-sharing processing results of the non-processed channel at other moments. The other moments can be adjacent moments to the current moment, such as moments that are at least one moment adjacent to the current moment.

[0072] The camera's pose information can be obtained by detecting the camera's position, such as by analyzing sensor data obtained from a pose sensor synchronized with the camera to obtain the camera's pose information at each moment. The pose sensor can be at least one of an odometer, an IMU (Inertial Measurement Unit), an image sensor, a GPS (Global Positioning System), or a Beidou system, though this disclosure is not limited thereto. The odometer can obtain wheel speed. The odometer and IMU can provide information such as the target device's motion, acceleration, and rotation. This information can be used to determine changes in the camera's position and pose in real time. The Global Positioning System provides the camera's position information by receiving signals from satellites. Using GPS, the camera's pose information, velocity, and other information can be obtained. The image sensor can capture image information. By analyzing the image information captured by the image sensor and extracting image features, changes in the camera's pose can be determined. Alternatively, using image feature extraction algorithms and optical flow algorithms, information such as the target device's movement and rotation can be obtained, thereby inferring changes in the camera's pose.

[0073] In some embodiments, during the step of determining the interpolation processing result for the non-processed channel at the current moment, information on the change in posture of the camera device corresponding to the non-processed channel between the current moment and the adjacent previous and / or subsequent moments can be obtained. Furthermore, the processing results of the non-processed channel at the adjacent previous and / or subsequent moments are obtained as adjacent processing results, where the adjacent processing results are obtained by processing the image frames of the non-processed channel at the adjacent moments using processing resources. Then, using the posture change information, the adjacent processing results are interpolated and shifted to obtain the interpolation processing result for the non-processed channel at the current moment.

[0074] Among them, the coordinate system of the camera device corresponding to the posture change information of the target device and the neighboring processing result can be obtained for coordinate conversion, and then the posture change information is used to obtain the predicted position of the target device in the neighboring processing result or the target object in the image, and the neighboring processing result is shifted in space to obtain the interpolation processing result of the non-processing channel at the current moment.

[0075] In some embodiments, the posture change information of the camera equipment of the non-processing channel or other channels can be used to replace or combine the posture change information of the camera equipment corresponding to the above-mentioned non-processing channel to interpolate and shift the adjacent processing results to obtain the interpolation processing results of the non-processing channel at the current moment. The interpolation processing method is similar to the above-mentioned interpolation method, and will not be repeated in this disclosure.

[0076] The above scheme, by utilizing the posture change information, interpolates and shifts the adjacent processing results to obtain the interpolation processing results of the non-processing channel at the current moment, which can make up for the missing information of the non-processing channel at the current moment, so that the integrity and accuracy of the processing results of the non-processing channel can be improved to a certain extent.

[0077] Returning to FIG4 , the above S13 may further include:

[0078] S132: The target processing result at the current moment is obtained by integrating the processing result at the current moment and the interpolation processing result.

[0079] The processing results corresponding to the first processing channel and the second processing channel at the current moment and the interpolation processing results corresponding to the non-processing channel can be integrated and projected into a common coordinate system to obtain the target processing result at the current moment.

[0080] As an example, referring to Figure 5 , multiple channels include a main channel CC and multiple auxiliary channels (e.g., SC1 and SC2). The processing times for main channel CC are T1, T2, T3, T4, etc., the processing times for auxiliary channel SC1 are T1, T3, etc., and the processing times for auxiliary channel SC2 are T2, T4, etc. At the current time T2, the first processing channel is main channel CC, the second processing channel is auxiliary channel SC2, and the non-processing channel is auxiliary channel SC1. For auxiliary channel SC1 at time T2, the processing result of auxiliary channel SC1 at time T1 can be interpolated and shifted using the position change information of the target device between times T1 and T2 to obtain the interpolated processing result of auxiliary channel SC1 at time T2. Similarly, the interpolated processing results of non-processing channels at other times can be obtained using the same method. Then, the processing result of main channel CC at time T2, the processing result corresponding to auxiliary channel SC2, and the interpolated processing result corresponding to auxiliary channel SC1 are combined and projected onto a common coordinate system to obtain the target processing result at time T2.

[0081] The above scheme obtains the target processing result at the current moment by combining the time-sharing processing results of the processing channel at the current moment and the interpolation processing results of the non-processing channel. While reducing the amount of image processing, the missing information can also be interpolated by time-sharing processing at other moments, so that the integrity and accuracy of the target processing result can be maintained to a certain extent.

[0082] The above-mentioned multi-channel video processing method can be used to perform object detection, route detection and other processing on the image frames of the video, especially for static objects such as runway areas and lane lines, and the image processing effect is good. Compared with independent processing of each camera device, it is easy to have insufficient or missing information related to the stability of the position and shape of static objects. The above-mentioned scheme of the present disclosure can use the posture information of the vehicle or the camera device for interpolation processing. Since the position and shape of static objects are relatively stable, the correct interpolation can be performed by using the time-sharing processing results of adjacent moments and the vehicle's posture change information, the information of the camera device, etc., to achieve accurate identification of static objects.

[0083] Furthermore, the reduced computational effort can reduce image processing power consumption, thereby improving power efficiency and extending battery life in vehicle-mounted systems. This reduction in computational effort can also increase image processing speed, enabling real-time object detection and route detection in vehicle driving scenarios, improving the performance of driver assistance systems. Due to the reduced computational effort, even cheaper and less powerful image processing devices can perform recognition processing. Consequently, the applicability of the disclosed multi-channel video processing method can be improved, and the system cost of autonomous or assisted vehicle driving can be reduced.

[0084] The above-mentioned solutions of the present disclosure can be applied to fields such as intelligent driving, automatic driving, assisted driving systems, mobile robot navigation, etc., but the present disclosure is not limited thereto.

[0085] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0086] With respect to the above embodiment, the present disclosure further provides a multi-channel video processing device, which can be used to implement the above multi-channel video processing method.

[0087] Please refer to Figure 6, which is a schematic diagram of the structure of an embodiment of a multi-channel video processing device of the present disclosure. The multi-channel video processing device 20 includes: an acquisition unit 21, a processing unit 22, and an integration unit 23. The acquisition unit 21, the processing unit 22, and the integration unit 23 are interconnected.

[0088] The acquisition unit 21 is used to acquire videos of multiple channels, wherein the multiple channels include at least one main channel and multiple auxiliary channels, and the videos of the main channel and the auxiliary channels are respectively captured by a first camera device and a second camera device of a predetermined scene.

[0089] The processing unit 22 is configured to process the image frames of the videos of the multiple channels using processing resources to obtain processing results of each channel, wherein the image frames of the videos of the multiple auxiliary channels are processed in a time-division processing manner.

[0090] The integration unit 23 is used to integrate the processing results of each channel to obtain the target processing result.

[0091] The above scheme obtains target videos of multiple channels, uses processing resources to process image frames of videos of multiple channels, and obtains processing results of each auxiliary channel. Among them, the image frames of the videos of the auxiliary channels are processed in a time-division manner, and the processing time of the videos of multiple auxiliary channels can be distributed in multiple different time periods. Then, the processing results of each channel are combined to obtain the target processing results. Compared with processing videos of all channels in the same scene at the same frequency, the amount of image processing calculations can be reduced, thereby reducing the consumption of processing resources and improving processing efficiency.

[0092] It should be understood that the units described in the apparatus 20 correspond to the steps in the method described with reference to Figures 2 to 5. Therefore, the operations and features described above for the method are also applicable to the apparatus 20 and the units contained therein, and will not be repeated here.

[0093] Please refer to Figure 7, which is a schematic diagram of the structure of an embodiment of a vehicle disclosed herein. The vehicle 30 includes: multiple cameras 31 and a processing module 32. The multiple cameras 31 include at least one first camera and multiple second cameras. The processing module 32 is used to provide processing resources and execute program data to implement any embodiment of the multi-channel video processing method described above.

[0094] In some embodiments, multiple camera devices 31 include a first camera device corresponding to a main channel and a second camera device corresponding to at least one auxiliary channel. The shooting field of view of the first camera device and the shooting field of view of each second camera device at least partially overlap. The first camera device is configured to capture video in the direction of travel of the vehicle, and multiple second camera devices are respectively configured to capture video of the side of the vehicle.

[0095] Regarding the above-mentioned embodiments, the present disclosure provides a computer device. Please refer to FIG8 , which is a schematic diagram of the structure of one embodiment of the computer device of the present disclosure. The computer device 40 includes a memory 41 and a processor 42 , wherein the memory 41 and the processor 42 are coupled to each other. The memory 41 stores program data, and the processor 42 is configured to execute the program data to implement the steps of any embodiment of the above-mentioned multi-channel video processing method.

[0096] In this embodiment, the processor 42 may also be referred to as a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip having signal processing capabilities. The processor 42 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor 42 may be any conventional processor.

[0097] The methods of the above embodiments can be implemented in the form of computer programs. Therefore, the present disclosure provides a computer-readable storage medium. See FIG9 , which is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present disclosure. The computer-readable storage medium 50 stores program data 51 capable of being executed by a processor. Program data 51 can be executed by the processor to implement the steps of any of the embodiments of the above-described multi-channel video processing method.

[0098] The computer-readable storage medium 50 in this embodiment can be a medium that can store program data 51, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or it can also be a server that stores the program data 51. The server can send the stored program data 51 to other devices for execution, or it can also execute the stored program data 51 itself.

[0099] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium. In other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0100] In some embodiments, the functions or modules included in each device or apparatus provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. The specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0101] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0102] In the several embodiments provided in the present disclosure, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0103] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present disclosure.

[0106] Obviously, those skilled in the art will appreciate that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a computer-readable storage medium and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any specific combination of hardware and software.

[0107] The above descriptions are merely embodiments of the present disclosure and are not intended to limit the patent scope of the present disclosure. Any equivalent structures or equivalent process transformations made using the contents of the present disclosure and the drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present disclosure.

Claims

1. A multi-channel video processing method, comprising: Acquire videos of multiple channels, where the multiple channels include at least one main channel and multiple auxiliary channels, where the videos of the main channel and the videos of the auxiliary channels are respectively acquired by using a first camera device and a second camera device of a predetermined scene; Using processing resources to process the image frames of the videos of the multiple channels to obtain processing results of each channel, wherein the image frames of the videos of the multiple auxiliary channels are processed in a time-division processing manner; The processing results of each channel are integrated to obtain the target processing result.

2. The method according to claim 1, wherein the processing of the image frames of the videos of the multiple channels by using the processing resources comprises: At each current moment in the process of processing the videos of the multiple channels, determining a processing channel and a non-processing channel at the current moment from among the multiple channels according to respective preset processing frame rates of the multiple channels; wherein the image frames of the videos of the non-processing channels are not processed at the current moment; The processing resources are used to process image frames of the video of the processing channel at the current moment.

3. The method according to claim 2, wherein the preset processing frame rate of the main channel is a first processing frame rate, and the preset processing frame rate of the auxiliary channel is a second processing frame rate; Determining a processing channel and a non-processing channel at a current moment from among the multiple channels according to the preset processing frame rates of the multiple channels respectively includes: determining, according to the first processing frame rate, whether the current moment is the processing moment of the main channel; in response to the current moment being the processing moment of the main channel, using the main channel as the first processing channel at the current moment; According to the second processing frame rate, determine whether the current moment is the processing moment of each auxiliary channel; use the auxiliary channel whose current moment is the processing moment as the second processing channel at the current moment; wherein the second processing frame rate is less than the first processing frame rate.

4. The method according to claim 3, The plurality of auxiliary channels are divided into a plurality of auxiliary channel groups, and within a processing period corresponding to the second processing frame rate, the processing times of the auxiliary channels in the same group are the same, and the processing times of the auxiliary channels in different groups do not overlap; The first processing frame rate is N times the second processing frame rate, and within a processing period corresponding to the second processing frame rate, the processing time of the main channel overlaps with the processing time of each auxiliary channel group once, and N is an integer greater than 1.

5. The method according to claim 3, further comprising: When the scene state of the predetermined scene meets a preset condition, a preset processing frame rate of at least one of the auxiliary channels is adjusted according to the scene state.

6. The method according to claim 5, wherein the predetermined scene includes a driving scene, the first camera device is used to capture a video of the vehicle's driving direction, the second camera device is used to capture a video of the vehicle's side, and the scene state includes a straight-moving state and a turning state of the vehicle; The adjusting the preset processing frame rate of at least one of the auxiliary channels according to the change of the scene state includes: When the vehicle is in a turning state, the preset processing frame rate of the auxiliary channel corresponding to the second camera device used to capture the video of the vehicle turning to one side is increased, and / or the preset processing frame rate of the auxiliary channel corresponding to the second camera device used to capture the video of the vehicle turning away from one side is reduced.

7. The method according to claim 2, wherein the step of synthesizing the processing results of each channel to obtain the target processing result comprises: For each current moment in the process of processing the videos of the multiple channels, determining, from each channel, a processing result of the processing channel at the current moment, and determining an interpolation processing result of the non-processing channel at the current moment, wherein the interpolation processing result is obtained by interpolating the position information of the camera device corresponding to the non-processing channel and the processing result of the non-processing channel at the previous moment and / or the subsequent moment of the current moment; The target processing result at the current moment is obtained by combining the processing result of the processing channel at the current moment and the interpolation processing result of the non-processing channel at the current moment.

8. The method according to claim 7, wherein determining the interpolation processing result of the non-processing channel at the current moment comprises: Acquire the position change information of the camera device corresponding to the non-processing channel between the current moment and the adjacent previous and / or subsequent moments; and obtaining the processing result of the non-processing channel at the adjacent preceding and / or subsequent moments as the adjacent processing result, wherein the adjacent processing result is a processing result obtained by processing the image frame of the non-processing channel at the adjacent moment by using the processing resource; The posture change information is used to interpolate and shift the adjacent processing results to obtain the interpolation processing results of the non-processing channel at the current moment.

9. The method according to any one of claims 1 to 8, The shooting fields of view of the first camera device and each of the second camera devices at least partially overlap.

10. A multi-channel video processing device, comprising: An acquisition unit, configured to acquire videos of a plurality of channels, wherein the plurality of channels include at least one main channel and a plurality of auxiliary channels, wherein the videos of the main channel and the videos of the auxiliary channels are acquired by respectively using a first camera device and a second camera device of a predetermined scene; A processing unit, used for processing the image frames of the videos of the plurality of channels by using processing resources to obtain processing results of each channel, wherein the image frames of the videos of the plurality of auxiliary channels are processed by time division processing; The synthesis unit is used to synthesize the processing results of each channel to obtain the target processing result.

11. A computer device, comprising a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is configured to execute the program data to implement the steps of the method according to any one of claims 1 to 9.

12. A vehicle comprising a plurality of camera devices and a processing module, wherein: The multiple camera devices include the first camera device and the second camera device, and the processing module is used to provide processing resources and to run program data to implement the steps of the method described in any one of claims 1 to 9.

13. The vehicle according to claim 12, The shooting field of view of the first camera device at least partially overlaps with the shooting field of view of each of the second camera devices. The first camera device is configured to capture video in the driving direction of the vehicle, and the plurality of second camera devices are respectively configured to capture video of the side of the vehicle. frequency.

14. A computer-readable storage medium storing program data that can be executed by a processor, wherein the program data is used to implement the steps of the method according to any one of claims 1 to 9.

15. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images