Image synchronization processing method and device, equipment, storage medium and program product
By setting a baseline frame rate and multiplier relationship in the extended reality device, the problem of increased processor load in synchronous processing is solved, achieving image data synchronization and improved processor efficiency.
Patent Information
- Application Number
- CN202310492621.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-04
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-05-04
AI Technical Summary
In existing extended reality devices, to achieve synchronized processing of images of parts such as the head and hands, the frame rate needs to be increased or decreased, which increases the processor's processing load and affects the product's performance.
By determining the changing attributes of the target objects, the image acquisition frame rate of the object with the slowest change is set as the baseline frame rate, and the frame rates of other target objects are set based on this to form a multiple relationship for image acquisition and synchronous processing.
Ensuring image data synchronization at different frame rates reduces the amount of processing power required by the processor, lowers the computational burden, and improves the overall efficiency of the processor.
Smart Images

Figure CN118900307A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image synchronization processing method, device, equipment, storage medium and program product. Background Art
[0002] At present, extended reality (XR) technology has gradually entered people's lives and various related technical fields. The corresponding augmented reality (AR) and virtual reality (VR) devices build a virtual three-dimensional visual environment by blocking people's vision, allowing people to participate in the environment and achieve an immersive experience. Current extended reality devices generally collect images and movements of the user's body, and in order to coordinate the images of different parts of the body, these images are generally processed synchronously.
[0003] However, with the development of image processing sophistication, for slower moving parts such as the head, a lower frame rate can be used to complete image sampling, but for faster moving parts such as the hands, a higher frame rate is required. Under the premise of synchronization, the relevant technology needs to increase or decrease the frame rate of both at the same time, which increases the processing burden and computing load of the processor, affecting the use effect of the product. Summary of the invention
[0004] In view of this, the present application proposes an image synchronization processing method, device, equipment, storage medium and program product, so as to reduce the number of processors to be processed, reduce the processing calculation burden and improve the overall efficiency of the processor while ensuring the synchronization of these groups of image data.
[0005] Based on the above purpose, the present application provides an image synchronization processing method, including:
[0006] Determine at least two target objects, and determine a change attribute of each of the target objects;
[0007] According to the change attribute, the image acquisition frame rate of the target object with the slowest change is set as a reference frame rate, and the image acquisition frame rates of other target objects are set based on the reference frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship;
[0008] Image acquisition is performed according to the image acquisition frame rate of each target object, and image synchronization processing is performed on the image corresponding to each target object based on the reference frame rate.
[0009] In some implementations, setting the image acquisition frame rate of other target objects based on the reference frame rate includes:
[0010] The at least two target objects are sorted according to the speed of change of the change attribute, and the image acquisition frame rate of the other target objects is set to a set multiple of the image acquisition frame rate of the target object that changes slower than the other target objects.
[0011] In some implementations, the performing image synchronization processing on the images corresponding to the target objects includes:
[0012] Determine a ratio between the image acquisition frame rate of each target object and the reference frame rate;
[0013] determining a processing period according to the reference frame rate, and dividing the processing period according to a maximum value among the ratios;
[0014] The image corresponding to each of the target objects is inserted into the divided processing cycle according to the corresponding ratio to perform the image synchronization.
[0015] In some implementations, dividing the processing period according to the maximum value of the ratios includes:
[0016] Determining the number of the target objects;
[0017] The time in the processing cycle is equally divided into divided areas whose number is the product of the maximum value of the ratio and the number.
[0018] In some implementations, inserting the image corresponding to each target object into the divided processing cycle according to the ratio includes:
[0019] Selecting an equal number of divided areas within the processing cycle in sequence and at intervals for each target object;
[0020] Determine, according to the ratio, a corresponding number of images in the images corresponding to each of the target objects as images to be inserted;
[0021] The image to be inserted is inserted into the corresponding divided area.
[0022] In some implementations, inserting the image to be inserted into the corresponding divided area includes:
[0023] In the current processing cycle, if any of the target objects has completed the insertion of the image to be inserted, the other divided areas corresponding to the target object and not inserted with the image to be inserted are set to empty.
[0024] In some implementations, selecting an equal number of divided areas within the processing cycle for each target object in sequence and at intervals includes:
[0025] Arranging the at least two target objects in sequence according to the image acquisition frame rate corresponding to each of the target objects;
[0026] Selecting the divided areas in sequence and at intervals within the processing cycle according to the arrangement order;
[0027] The inserting the image to be inserted into the corresponding divided area comprises:
[0028] According to the arrangement order, the corresponding images to be inserted are inserted into the divided areas in sequence and at intervals from each other.
[0029] In some implementations, inserting the to-be-inserted images into the divided areas sequentially and spaced apart from each other comprises:
[0030] According to the image acquisition frame rate corresponding to each target object, the divided area to be inserted corresponding to each target object is selected in the current processing cycle, so that the intervals between the images to be inserted are kept consistent between multiple processing cycles.
[0031] In some implementations, the images corresponding to each of the target objects are arranged in sequence according to acquisition time.
[0032] Based on the same concept, the present application also provides an image synchronization processing device, including:
[0033] A determination module, used to determine at least two target objects and determine a change attribute of each of the target objects;
[0034] A setting module, for setting the image acquisition frame rate of the target object with the slowest change as a reference frame rate according to the change attribute, and setting the image acquisition frame rates of other target objects based on the reference frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship;
[0035] The processing module is used to perform image acquisition according to the image acquisition frame rate of each target object, and perform image synchronization processing on the image corresponding to each target object based on the reference frame rate.
[0036] Based on the same concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.
[0037] Based on the same concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement any of the methods described above.
[0038] Based on the same concept, the present application also provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute any of the above methods.
[0039] From the above, it can be seen that the present application provides an image synchronization processing method, device, equipment, storage medium and program product, including: determining at least two target objects, and determining the change attributes of each target object; according to the change attributes, the image acquisition frame rate of the target object with the slowest change is set as the reference frame rate, and the image acquisition frame rates of other target objects are set based on the reference frame rate, so that the image acquisition frame rates of each target object form a multiple relationship; image acquisition is performed according to the image acquisition frame rate of each target object, and image synchronization processing is performed on the images corresponding to each target object based on the reference frame rate. When processing at least two target objects, the present application first determines the speed of the change attribute, sets the acquisition frame rate of the target object with the slowest change as the reference frame rate, and then determines the acquisition frame rate of other target objects based on this, and finally ensures the synchronization of the image data of these target objects when the acquisition frame rates are different, thereby reducing the number of processors to be processed, reducing the processing calculation burden, and improving the overall efficiency of the processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flowchart of an image synchronization processing method provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of synchronously processing head images, hand images, and leg images with different frame rates in a specific embodiment provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of synchronously processing head images and hand images with different frame rates in a specific embodiment provided in an embodiment of the present application;
[0044] Figure 4 A schematic diagram of the structure of an image synchronization processing device provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of this specification more clear, this specification is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0047] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0048] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.
[0049] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0050] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.
[0051] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0052] In order to make the purpose, technical solutions and advantages of this specification more clear, this specification is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0053] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements, objects or method steps appearing before the word cover the elements, objects or method steps listed after the word and their equivalents, without excluding other elements, objects or method steps. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0054] As mentioned in the background technology section, XR (Extended Reality) technology refers to an environment that combines reality and virtuality and allows human-computer interaction through computer technology and wearable devices. XR technology can be regarded as an umbrella term that includes VR (virtual reality), AR (augmented reality), MR (mixed reality) and other new immersive technologies that may emerge due to technological advances. XR is a general term for immersive virtual and reality fusion technology, from which different technical branches such as VR, AR, and MR are derived. VR can immerse people completely in a virtual environment; AR can generate a world with superimposed virtual content, but cannot interact with the real environment; MR is a mixture of virtual and reality, which can create virtual objects that can interact with the real environment. At present, extended reality (XR) technology has gradually entered people's lives and various related technical fields. Technologies such as augmented reality (AR) and virtual reality (VR) in XR technology have been applied to shopping, games, entertainment, sports and fitness, etc. On the extended reality device, there will be multiple cameras to identify the position of the head, handles, gestures, etc.; among them, for the head and other positions with slower movement speed, a lower frame rate is used for image sampling, such as 30fps, to clearly reflect the movement changes of the head; for the hands and other positions with faster movement speed, a higher frame rate is generally required for image sampling to clearly reflect the movement changes, such as 60fps. However, in order to achieve synchronous processing, if the frame rate of the head and hand is doubled, the amount of data that the main processor needs to process will double. For example, now the head and hand need to process 30 frames of images per second, and the main processor needs 60 frames of images per second; if it is increased to 60 frames each, the main processor needs to process 120 frames of images per second, but for data with slow movement speed such as the head, 30 frames per second can actually meet the requirements. This has led to an increase in the processing burden and computing load of the processor under the current synchronization premise, affecting the use effect of the product.
[0055] In view of the above actual situation, the embodiment of the present application provides an image synchronization processing solution. When processing at least two target objects, the present application first determines the speed of attribute change, sets the acquisition frame rate of the slowest changing target object as the reference frame rate, and then determines the acquisition frame rate of other target objects based on this, and finally ensures the synchronization of image data of these target objects when the acquisition frame rates are different, thereby reducing the number of processors to be processed, reducing the processing calculation burden, and improving the overall efficiency of the processor.
[0056] like Figure 1 FIG. 1 is a flow chart of an image synchronization processing method proposed in this application. Specifically, it includes:
[0057] Step 101: determine at least two target objects, and determine a change attribute of each of the target objects.
[0058] In some embodiments, the target object may be a target that requires image synchronization processing, and they may be interrelated, for example, if the XR device needs to synchronize the user's head image and hand image, then the target object is the user's head and hand; they may also be unrelated to each other, but image synchronization is required in specific scenarios, such as the synchronization of bicycle images and car images on the same section of road.
[0059] After that, the change attributes of these target objects are determined. Here, the change attributes can be the attributes of the overall movement or partial movement of a person or object, such as the position movement of the head / handle / gesture, the movement of a bicycle, etc.; they can also be the attributes of the change of local features, such as changes in facial expressions, gestures, etc. The specific ways of these changes may be different, but each change will correspond to an attribute of the speed of change, which can be understood as the change speed here. The change speed here can be generated by comparing with each other, or by comparing with a reference value. Taking the user's head and hand as an example, the change speed of the head is significantly less than that of the hand when compared between the two, or the movement speed can be calculated separately to determine the speed of change of the two. These can be measured and calculated by sensors, etc. Of course, a table for recording the change speed can also be set according to historical experience to record the change speed of some common things. The change speed here can generally be relative, that is, the target objects that need image synchronization are compared to determine the speed of change between them.
[0060] Step 102, according to the change attribute, the image acquisition frame rate of the target object with the slowest change is set as the base frame rate, and the image acquisition frame rates of other target objects are set based on the base frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship.
[0061] In some embodiments, the image acquisition frame rate is generally expressed as the frame rate data for image acquisition. The frame rate data (FPS, Frame Per Second) generally refers to the number of frames transmitted per second in an animation or video. For example, 30fps means that 30 images are acquired per second for processing and display. Afterwards, the reference frame rate can be a pre-set frame rate value, which can be specifically set according to different specific scenes or different facial objects. For example, in indoor activity scenes, for images with a slower changing speed, such as the image of the user's head, only 30fps is required to clearly reflect the corresponding image; in other scenes, such as scenes of image capture on the road, although the bicycle is a relatively slow changing image, it still requires 60fps to clearly reflect the corresponding image.
[0062] After that, the image acquisition frame rate of the target object is set to a multiple relationship. For example, for the three target objects of head, hand and leg, the head changes the slowest, followed by the leg, and the hand the highest. Then, the image acquisition frame rate of the head is set to the base frame rate, such as 30fps; then, the leg is set to a multiple relationship with the head, such as 2 times, and the image acquisition frame rate of the leg is 60fps; similarly, the hand can be a multiple relationship of both the leg and the head, so the hand can be 4 times that of the head and 2 times that of the leg, that is, the image acquisition frame rate of the hand is 120fps. Of course, the specific multiple can be set according to the specific scene.
[0063] Step 103 , performing image acquisition according to the image acquisition frame rate of each target object, and performing image synchronization processing on the image corresponding to each target object based on the reference frame rate.
[0064] In some embodiments, images of these target objects are collected according to the image acquisition frame rate determined for each target object. Each group of target object images collected here can be the image data of the target object acquired per second, for example, a group of image data for the head (30fps) is 30 head images acquired within 1 second, a group of image data for the hand (60fps) is 60 hand images acquired within 1 second, and so on. Each group of image data can be arranged according to the acquisition time of the image. Here, since the multiple relationship between the image acquisition frame rates of the target objects has been set in step 102, the number of images acquired here is related to the image acquisition frame rate, for example, if the frame rate of the head image is 30fps, 30 images are collected per second; if the frame rate of the hand image is 120fps, 120 images are collected per second; if the frame rate of the leg image is 60fps, 60 images are collected per second, and so on.
[0065] Afterwards, based on the reference frame rate, the images corresponding to each of the target objects are synchronously processed. In order to ensure the synchronization requirements here, it is necessary to determine that at least one image of each group of image data is processed within a processing cycle, so it is necessary to ensure that the slowest target object has an image processed within a cycle based on the reference frame rate. For example, the head has the lowest change speed among all target objects, and the reference frame rate is 30fps, which corresponds to processing one image every 1 / 30s. According to this solution, the processing cycle is 1 / 30s, and one head image can be processed within one cycle. Afterwards, since the acquisition frame rate of other target objects is in a multiple relationship with the target object with the lowest frame rate change, a cycle can be divided according to this multiple, and the images of the corresponding target objects of the corresponding multiple can be processed within one cycle, so as to achieve the synchronization effect.
[0066] In a specific embodiment, the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps. The multiples between the hand and the head are 4, the multiples between the leg and the head are 2, and the multiples between the hand and the leg are 2. The processing cycle is divided according to the maximum value 4, and the division method can be based on a specific multiple of the maximum value. For example, if there are 3 groups of data here, the processing cycle is divided into 12 parts, which is the product of the maximum value 4 and the number of groups 3, that is, the processing cycle of 1 / 30s is divided into 12 divided areas of 1 / 360s, for a total of 12 parts, such as Figure 2 After that, one image is selected from the head image data according to the magnification, four images are selected from the hand image data according to the magnification, and two images are selected from the leg image data according to the magnification, and are placed in the divided processing cycle, as shown in FIG. Figure 2 As shown, the synchronization effect between the groups of image data is ensured while ensuring the processing frame rate data corresponding to each group of image data.
[0067] In another specific embodiment, according to the solution of the above embodiment, for head images, the image acquisition frame rate is 30fps, and for hand images, the image acquisition frame rate is 60fps. Figure 3 As shown, the processing cycle is 1 / 30s; the processing cycle is divided according to the maximum value of the multiple 2. Here, there are 2 groups of data, so the processing cycle is divided into 4 parts, each of which is 1 / 120s; then according to the multiple, 1 image is selected from the head image and 2 images are selected from the hand image, and put into the divided processing cycle. In this way, while maintaining the synchronization effect, the image processing frame rate only needs 90 frames per second. Compared with the scheme in which both need to change synchronously in the related scheme (either 30 frames or 60 frames), it reduces the calculation load of 30 frames, reduces the number of processors to be processed, reduces the processing calculation burden, and improves the overall efficiency of the processor.
[0068] Finally, the processing cycle after the processing is completed can be output, and then the corresponding image processing can be performed based on this processing cycle. At the same time, it is also used to store, display, use or reprocess the processing cycle. Among them, according to different application scenarios and implementation needs, the specific output method for the processing cycle can be flexibly selected.
[0069] For example, for an application scenario where the method of this embodiment is executed on a single device, the processing cycle can be directly applied to the image processing of the current device and output in an execution manner, so that the image display of the current device is processed and displayed according to the processing cycle.
[0070] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the processing cycle can be sent to other preset devices as receivers in the system, that is, synchronization terminals, through any data communication method (wired connection, NFC, Bluetooth, wifi, cellular mobile network, etc.), so that the synchronization terminal can perform subsequent processing on it. Optionally, the synchronization terminal can be a preset server, which is generally set up in the cloud as a data processing and storage center, which can store and distribute the processing cycle; wherein the recipient of the distribution is the terminal device, and the holder or operator of these terminal devices can be other smart devices of the current user, the maintainer of the smart device, the producer of the related smart device, etc.
[0071] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the processing cycle can be sent directly to a preset terminal device through any data communication method, and the terminal device can be one or more of the ones listed in the preceding paragraphs.
[0072] From the above, it can be seen that the image synchronization processing method provided by the present application includes: determining at least two target objects, and determining the change attributes of each target object; according to the change attributes, setting the image acquisition frame rate of the target object with the slowest change as the reference frame rate, and setting the image acquisition frame rates of other target objects based on the reference frame rate, so that the image acquisition frame rates of each target object form a multiple relationship; performing image acquisition according to the image acquisition frame rate of each target object, and performing image synchronization processing on the images corresponding to each target object based on the reference frame rate. When processing at least two target objects, the present application first determines the speed of the change attributes, sets the acquisition frame rate of the target object with the slowest change as the reference frame rate, and then determines the acquisition frame rate of other target objects based on this, and finally ensures the synchronization of the image data of these target objects when the acquisition frame rates are different, thereby reducing the number of processors to be processed, reducing the processing calculation burden, and improving the overall efficiency of the processor.
[0073] In some embodiments, setting the image acquisition frame rate of other target objects based on the benchmark frame rate includes: sorting the at least two target objects according to the speed of change of the changing attribute, and setting the image acquisition frame rate of the other target objects to a set multiple of the image acquisition frame rate of the target object that changes slower than the other target objects.
[0074] In some embodiments, in order to quickly set the image acquisition frame rate of each target object, it can be sorted according to the speed of change of the change attribute of different target objects, and the sorting can be from fast to slow or from slow to fast, wherein the slowest changing image acquisition frame rate is set as the base frame rate, and then the second to last is set as the set multiple of the base frame rate, and the third to last is set as the set multiple of the second to last, and so on, so that the image acquisition frame rates of each target object are in a multiple relationship, so as to facilitate the subsequent image synchronization processing. Moreover, in this embodiment, the image acquisition frame rate corresponding to the target object with a fast change is in a multiple relationship with the image acquisition frame rate corresponding to the target object with a slower change, for example, the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps, and the frame rate of the hand image is 4 times that of the head image and 2 times that of the leg image. It can be seen that if a target object corresponds to multiple other target objects, the set multiples here can also be multiple.
[0075] In some embodiments, the image synchronization processing of the images corresponding to each of the target objects includes: determining the ratio between the image acquisition frame rate of each of the target objects and the reference frame rate; determining a processing period based on the reference frame rate, and dividing the processing period according to the maximum value in the ratio; inserting the image corresponding to each of the target objects into the divided processing period according to the corresponding ratio to perform the image synchronization.
[0076] In some embodiments, in order to ensure synchronization requirements, it is necessary to ensure that at least one image of each target object is processed within a processing cycle. Here, the processing time corresponding to the benchmark frame rate can be used as the processing cycle. For example, the benchmark frame rate is 30fps, which corresponds to processing one image every 1 / 30s. According to this solution, the processing cycle is 1 / 30s.
[0077] Afterwards, the processing cycle is determined by the reference frame rate, and the processing cycle is divided according to the maximum value of the ratio. For example, if the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps, then the ratio of the hand to the head is 4, the ratio of the leg to the head is 2, and the ratio of the hand to the leg is 2. Then the processing cycle is divided according to the maximum value 4, and the division method can be based on a specific multiple of the maximum value. For example, if there are 3 groups of data here, the processing cycle is divided into 12 parts, which is the product of the maximum value 4 and the number of groups 3, that is, the processing cycle of 1 / 30s is divided into 12 divided areas of 1 / 360s, for a total of 12 parts, such as Figure 2 shown.
[0078] Finally, the image corresponding to each target object is inserted into the divided processing cycle according to the ratio to perform the image synchronization. According to the previous example, one image is selected from the head image data according to the ratio, four images are selected from the hand image data according to the ratio, and two images are selected from the leg image data according to the ratio, and placed in the divided processing cycle, such as Figure 2 As shown, the synchronization effect between the groups of image data is ensured while ensuring the processing frame rate data corresponding to each group of image data.
[0079] In some embodiments, dividing the processing cycle according to the maximum value in the ratio includes: determining the number of the target objects; dividing the time in the processing cycle into equal parts of the divided areas whose number is the product of the maximum value in the ratio and the number.
[0080] In some embodiments, in order to meet the frame rate requirements of each set of image data while maintaining synchronization, when dividing the processing cycle, it is necessary to meet the frame rate requirements of each target object with a high frame rate. Then, the specific number of target objects can be determined. For example, the head and the hand are two target objects, and the head, the hand and the leg are three target objects. Then, the number is multiplied by the maximum value in the ratio to determine the number of divided areas, and the divided areas are the divided areas. For example, Figure 2 As shown, there are 3 groups of data, the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps. The processing period is divided into 12 parts, which is the product of the maximum value 4 in the frame rate ratio and the number of groups 3. That is, the processing period of 1 / 30s is divided into 12 divided areas of 1 / 360s each.
[0081] In some embodiments, inserting the image corresponding to each of the target objects into the divided processing cycle according to the ratio includes: selecting an equal number of divided areas within the processing cycle for each of the target objects in sequence; determining a corresponding number of images corresponding to each of the target objects as images to be inserted according to the ratio; and inserting the images to be inserted into the corresponding divided areas.
[0082] In some embodiments, Figure 2 or Figure 3 As shown, a corresponding number of partition areas can be selected for each target object. Since the partition area is determined by the product of the number of groups and the maximum value in the ratio, each target object can certainly select the partition area with the maximum value in the ratio, such as Figure 2 In , the head image, hand image and leg image correspond to 4 divided areas respectively; Figure 3 In , the head image and the hand image correspond to two divided areas respectively. Then, according to the ratio, it is determined how many images each target object needs to insert in one cycle. These images that need to be inserted are the images to be inserted. Figure 2 In , the head image needs to be inserted into 1 image, the hand image needs to be inserted into 4 images, and the leg image needs to be inserted into 2 images; Figure 3 In the example, the head image needs to be inserted into one image and the hand image needs to be inserted into two images. Finally, the selected images are inserted into the corresponding divided areas. Only one image is inserted into one of the divided areas.
[0083] In some embodiments, inserting the image to be inserted in the corresponding divided area includes: within the current processing cycle, if any of the target objects has completed the insertion of the image to be inserted, setting the other divided areas corresponding to the target object that have not inserted the image to be inserted to empty.
[0084] In some embodiments, Figure 2 or Figure 3 As shown, if any group of image data has completed the image insertion within the processing cycle, the other divided areas corresponding to the group of image data that have not yet been inserted with the image to be inserted are set to empty. Figure 2 In the example, the head image corresponds to four divided areas, but there is only one image to be inserted. After the image to be inserted is inserted into one divided area, the other divided areas are set to be empty. In each processing cycle, the order or rule for inserting the images is not specifically limited here, and it only needs to insert a corresponding number of images in one processing cycle.
[0085] In some embodiments, selecting the divided areas within the processing cycle at equal intervals in sequence for each of the target objects comprises: arranging the at least two target objects in sequence according to the image acquisition frame rate corresponding to each of the target objects; selecting the divided areas in sequence at intervals in the processing cycle according to the arrangement order; inserting the image to be inserted in the corresponding divided area comprises: inserting the corresponding image to be inserted in the divided area in sequence at intervals from each other according to the arrangement order.
[0086] In some embodiments, in order to facilitate the management of images of various target objects, the images can be sorted according to the size of the image acquisition frame rate corresponding to each target object, and then the divided areas can be selected in sequence according to the sorting order. For example, if the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps, then they can be sorted into head images, leg images, and hand images. When selecting the divided areas, the first divided area is the area of the head image, the second divided area is the area of the leg image, the third divided area is the area of the hand image, the fourth divided area is the area of the head image, and so on. Therefore, when inserting images, the images to be inserted are also inserted according to the arrangement order.
[0087] In some embodiments, inserting the images to be inserted in the divided areas in sequence and at intervals from each other includes: according to the image acquisition frame rate corresponding to each target object, selecting the divided area where the images to be inserted corresponding to each target object need to be inserted within the current processing cycle, so that the intervals between the images to be inserted are kept consistent between multiple processing cycles.
[0088] In some embodiments, similar to the above embodiments, if any group of image data has completed the insertion of images within the processing cycle, the other divided areas corresponding to the group of image data that have not yet inserted the images to be inserted are set to empty. In this embodiment, since each target object has a corresponding image acquisition frame rate, in order to facilitate processing and image insertion, the number of insertions can be guaranteed within a processing cycle while ensuring that the image acquisition intervals of the same target object between processing cycles are consistent, thereby ensuring that image processing at different frame rates can be completed quickly without adjusting the frame rate. As shown in the attached figure, Figure 2 As shown, the frame rate of the head image is 30fps, the frame rate of the hand image is 120fps, and the frame rate of the leg image is 60fps. Between processing cycles, the spacing between the head image, hand image, and leg image is also the same. In this way, within one second, the frame rate of the head image still meets 30fps, the hand image also meets 120fps, and the leg image also meets 60fps.
[0089] In some embodiments, the images corresponding to each of the target objects are arranged in sequence according to acquisition time.
[0090] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of the embodiment of the present application can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0091] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0092] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an image synchronization processing device.
[0093] refer to Figure 4 , the image synchronization processing device comprises:
[0094] A determination module 210, configured to determine at least two target objects and determine a change attribute of each of the target objects;
[0095] A setting module 220, configured to set the image acquisition frame rate of the target object with the slowest change as a reference frame rate according to the change attribute, and to set the image acquisition frame rates of other target objects based on the reference frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship;
[0096] The processing module 230 is used to perform image acquisition according to the image acquisition frame rate of each target object, and perform image synchronization processing on the image corresponding to each target object based on the reference frame rate.
[0097] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0098] The device of the above embodiment is used to implement the corresponding image synchronization processing method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0099] In some embodiments, the setting module 220 is further used to:
[0100] The at least two target objects are sorted according to the speed of change of the change attribute, and the image acquisition frame rate of the other target objects is set to a set multiple of the image acquisition frame rate of the target object that changes slower than the other target objects.
[0101] In some embodiments, the processing module 230 is further configured to:
[0102] Determine a ratio between the image acquisition frame rate of each target object and the reference frame rate;
[0103] determining a processing period according to the reference frame rate, and dividing the processing period according to a maximum value among the ratios;
[0104] The image corresponding to each of the target objects is inserted into the divided processing cycle according to the corresponding ratio to perform the image synchronization.
[0105] In some embodiments, the processing module 230 is further configured to:
[0106] Determining the number of the target objects;
[0107] The time in the processing cycle is equally divided into divided areas whose number is the product of the maximum value of the ratio and the number.
[0108] In some embodiments, the processing module 230 is further configured to:
[0109] Selecting an equal number of divided areas within the processing cycle in sequence and at intervals for each target object;
[0110] Determine, according to the ratio, a corresponding number of images in the images corresponding to each of the target objects as images to be inserted;
[0111] The image to be inserted is inserted into the corresponding divided area.
[0112] In some embodiments, the processing module 230 is further configured to:
[0113] In the current processing cycle, if any of the target objects has completed the insertion of the image to be inserted, the other divided areas corresponding to the target object and not inserted with the image to be inserted are set to empty.
[0114] In some embodiments, the processing module 230 is further configured to:
[0115] Arranging the at least two target objects in sequence according to the image acquisition frame rate corresponding to each of the target objects;
[0116] Selecting the divided areas in sequence and at intervals within the processing cycle according to the arrangement order;
[0117] The processing module 230 is further used for:
[0118] According to the arrangement order, the corresponding images to be inserted are inserted into the divided areas in sequence and at intervals from each other.
[0119] In some embodiments, the processing module 230 is further configured to:
[0120] According to the image acquisition frame rate corresponding to each target object, the divided area to be inserted of the image to be inserted corresponding to each target object is selected in the current processing cycle, so that the intervals between the images to be inserted are kept consistent between multiple processing cycles.
[0121] In some embodiments, the images corresponding to each of the target objects are arranged in sequence according to acquisition time.
[0122] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image synchronization processing method as described in any of the above embodiments is implemented.
[0123] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0124] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0125] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0126] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0127] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0128] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0129] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0130] The electronic device of the above embodiment is used to implement the corresponding image synchronization processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0131] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the image synchronization processing method described in any of the above embodiments.
[0132] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0133] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the image synchronization processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0134] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the image synchronization processing method. Corresponding to the execution subject corresponding to each step in each embodiment of the image synchronization processing method, the processor that executes the corresponding step can belong to the corresponding execution subject.
[0135] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the image synchronization processing method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0136] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0137] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0138] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0139] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. An image synchronization processing method, characterized in that: include: Determine at least two target objects, and determine a change attribute of each of the target objects; According to the change attribute, the image acquisition frame rate of the target object with the slowest change is set as a reference frame rate, and the image acquisition frame rates of other target objects are set based on the reference frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship; Image acquisition is performed according to the image acquisition frame rate of each target object, and image synchronization processing is performed on the images corresponding to each target object based on the reference frame rate.
2. The method according to claim 1, characterized in that The step of setting the image acquisition frame rate of other target objects based on the reference frame rate includes: The at least two target objects are sorted according to the speed of change of the change attribute, and the image acquisition frame rate of the other target objects is set to a set multiple of the image acquisition frame rate of the target object that changes slower than the other target objects.
3. The method according to claim 1, characterized in that The performing image synchronization processing on the images corresponding to the target objects includes: Determine a ratio between the image acquisition frame rate of each target object and the reference frame rate; determining a processing period according to the reference frame rate, and dividing the processing period according to a maximum value among the ratios; The image corresponding to each of the target objects is inserted into the divided processing cycle according to the corresponding ratio to perform the image synchronization.
4. The method according to claim 3, characterized in that The dividing the processing period according to the maximum value of the ratios comprises: Determining the number of the target objects; The time in the processing cycle is equally divided into divided areas whose number is the product of the maximum value of the ratio and the number.
5. The method according to claim 4, characterized in that Inserting the image corresponding to each target object into the divided processing cycle according to the ratio comprises: Selecting an equal number of divided areas within the processing cycle in sequence and at intervals for each target object; Determine, according to the ratio, a corresponding number of images in the images corresponding to each of the target objects as images to be inserted; The image to be inserted is inserted into the corresponding divided area.
6. The method according to claim 5, characterized in that The inserting the image to be inserted into the corresponding divided area comprises: In the current processing cycle, if any of the target objects has completed the insertion of the image to be inserted, the other divided areas corresponding to the target object and not inserted with the image to be inserted are set to empty.
7. The method according to claim 5, characterized in that The step of selecting an equal number of divided areas within the processing cycle for each target object in sequence and at intervals includes: Arranging the at least two target objects in sequence according to the image acquisition frame rate corresponding to each of the target objects; Selecting the divided areas in sequence and at intervals within the processing cycle according to the arrangement order; The inserting the image to be inserted into the corresponding divided area comprises: According to the arrangement order, the corresponding images to be inserted are inserted into the divided areas in sequence and at intervals from each other.
8. The method according to claim 7, characterized in that Inserting the to-be-inserted images into the divided areas sequentially and spaced apart from each other comprises: According to the image acquisition frame rate corresponding to each target object, the divided area to be inserted corresponding to each target object is selected in the current processing cycle, so that the intervals between the images to be inserted are kept consistent between multiple processing cycles.
9. The method according to claim 1, characterized in that: The images corresponding to each target object are arranged in sequence according to the acquisition time.
10. An image synchronization processing device, characterized in that: include: A determination module, used to determine at least two target objects and determine a change attribute of each of the target objects; A setting module, for setting the image acquisition frame rate of the target object with the slowest change as a reference frame rate according to the change attribute, and setting the image acquisition frame rates of other target objects based on the reference frame rate, so that the image acquisition frame rates of the target objects form a multiple relationship; The processing module is used to perform image acquisition according to the image acquisition frame rate of each target object, and perform image synchronization processing on the image corresponding to each target object based on the reference frame rate.
11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 9 is implemented.
12. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to implement the method according to any one of claims 1 to 9.
13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Novel automobile and automobile image processing system thereof
CN111093041A
Target track generation method and device, electronic equipment and medium
CN114066974A
Image processing apparatus and image processing method
JP2020092394A
Playback apparatus
JP2021029052A
Bifocal camera temperature measurement method based on image processing, and related device
WO2021232587A1
