Image processing methods, equipment and media

By identifying and processing keyframe images captured by AI glasses, the problems of wasted image storage space and network resource consumption in AI glasses are solved, achieving efficient storage and transmission.

CN123075183APending Publication Date: 2026-11-10GOERTEK INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611006906.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-07
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

AI glasses suffer from wasted storage space and high network resource consumption when acquiring images, because some images have similar content, leading to unnecessary storage and transmission.

Method used

By acquiring metadata from the image sequence, keyframe images are identified and differentially compressed, non-keyframe images are discarded, and different compression parameters are used to process the keyframe images.

Benefits of technology

It effectively avoids wasting storage space and network resources, improves storage and transmission efficiency, and ensures the visual quality of critical images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123075183A_ABST
    Figure CN123075183A_ABST
Patent Text Reader

Abstract

This disclosure provides an image processing method, apparatus, and medium, relating to the field of image processing technology. The method includes: acquiring an image sequence, comprising multiple first images sequentially acquired by an image acquisition device and metadata for each first image. The metadata includes at least one of: feature values ​​of the motion mode of the image acquisition device when acquiring the first image, the image recording value of the first image, and the image subject identifier of the first image; for each first image, identifying whether the first image is a keyframe image based on its metadata; discarding the first image if it is not a keyframe image, and retaining the first image if it is a keyframe image. This method allows only first images with retention value to be retained. This avoids the problem of wasted storage space or high network resource consumption caused by retaining first images without retention value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, and medium. Background Technology

[0002] With the development of technology and economy, AI glasses are widely used in long-term recording scenarios such as security patrol, first-person perspective recording, and life logs.

[0003] Currently, AI glasses store the images they capture. However, some images captured by AI glasses have certain content similarities, which leads to wasted storage space when storing the images or high network resource consumption when transmitting them. Summary of the Invention

[0004] One object of this disclosure is to provide a new technical solution for image processing.

[0005] According to a first aspect of this disclosure, an image processing method is provided, applied to an image processing device, the method comprising: An image sequence is acquired, the image sequence including multiple frames of first images acquired sequentially by an image acquisition device and metadata of each first image, the metadata including at least one of the following: feature value of the motion mode of the image acquisition device when the first image is acquired, image recording value of the first image, and image subject identifier of the first image; For each of the first images, identify whether the first image is a keyframe image based on the metadata of the first image; If the first image is not a keyframe image, discard the first image; if the first image is a keyframe image, retain the first image.

[0006] Optionally, identifying whether the first image is a keyframe image based on the metadata of the first image includes: If the metadata of the first image satisfies any one of the first to third items below, the first image is determined to be a first keyframe image, and the keyframe image includes the first keyframe image. The first step involves comparing the image subject identifier corresponding to the first image with the image subject identifier of the second image to determine that the image subject identifier corresponding to the first image and the image subject identifier of the second image are different, and the second image is the frame image preceding the first image in the image sequence. The second step is to compare the feature values ​​of the motion pattern of the first image with the feature values ​​of the motion pattern of the second image to determine that the feature values ​​of the motion pattern of the first image are different from the feature values ​​of the motion pattern of the second image. Thirdly, the image recording value of the first image is greater than the first preset image recording value.

[0007] Optionally, if the metadata of the first image does not satisfy the first, second, and third items, the method further includes: Based on the first image and the third image, determine the image similarity between the first image and the third image, wherein the third image is the first keyframe image that is temporally closest to the first image; If the image similarity is less than or equal to a preset similarity, the first image is determined to be the second keyframe image, and the keyframe image includes the second keyframe image.

[0008] Optionally, retaining the first image when the first image is a keyframe image includes: If the first image is a first keyframe image, the first image is compressed according to the first compression parameters; If the first image is a second keyframe image, the first image is compressed according to the second compression parameters; The compression quality of the first compression parameter is higher than that of the second compression parameter.

[0009] Optionally, before compressing the first image according to the second compression parameters when the first image is a second keyframe image, the method further includes: Obtain the current remaining storage capacity value of the image processing device; When the first image is a second keyframe image, compressing the first image according to the second compression parameters includes: If the current remaining storage capacity value is greater than or equal to the preset storage capacity value and the first image is a second keyframe image, the first image is compressed according to the second compression parameters. The method further includes: When the current remaining storage capacity is less than the preset storage capacity and the first image is a second keyframe image, the first image is compressed according to the third compression parameter, where the compression quality of the second compression parameter is higher than the compression quality of the third compression parameter.

[0010] Optionally, the image sequence is obtained according to the following steps: Acquire the current motion data collected by the motion sensor in the image acquisition device within a preset time period before the current moment; Based on the current motion data, determine the current image recording value and the current change in viewing angle of the image sensor in the image acquisition device; When the current image recording value is greater than or equal to the second preset image recording value, or when the current viewpoint change is greater than or equal to the preset viewpoint change, the image sensor is controlled to acquire an image to obtain a first image, wherein the first preset image recording value is greater than the second preset image recording value. Extract the metadata of the first image; An image sequence is generated based on the first image and its corresponding metadata acquired sequentially by the image acquisition device.

[0011] Optionally, the current motion data includes current angular velocity data in the image plane width direction, and determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Integrating the current angular velocity data yields the current yaw angle change in the coordinate system of the motion sensor. Convert the current yaw angle change in the coordinate system of the motion sensor to the current yaw angle change in the coordinate system of the image sensor. Based on the current change in yaw angle in the image sensor coordinate system, determine the current pixel displacement in the image plane width direction; Based on the current pixel displacement and the image plane width value corresponding to the image sensor, the percentage change in the current viewing angle in the image plane width direction is determined as the current viewing angle change of the image sensor.

[0012] Optionally, determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Based on the current motion data, determine the current feature value of at least one motion feature of the image acquisition device; The current image recording value is determined based on the current feature value of the at least one motion feature.

[0013] According to a second aspect of this disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform the method as described in any one of the first aspects.

[0014] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0015] This disclosure provides an image processing method applied to an image processing device. The method includes: acquiring an image sequence, the image sequence including multiple first images sequentially acquired by an image acquisition device and metadata for each first image, the metadata including at least one of: feature values ​​of the motion mode of the image acquisition device when acquiring the first image, the image recording value of the first image, and the image subject identifier of the first image; for each first image, identifying whether the first image is a keyframe image based on the metadata of the first image; discarding the first image if it is not a keyframe image, and retaining the first image if it is a keyframe image. This method enables the retention of only first images with retention value. This avoids the problem of wasted storage space or high network resource consumption caused by retaining first images without retention value.

[0016] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0018] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this disclosure. Detailed Implementation

[0019] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] This disclosure provides an image processing method applied to an image processing device. The image processing device can be an image acquisition device, or it can be another device independent of the image acquisition device, such as a computer or server. An example image acquisition device can be AI glasses.

[0025] like Figure 1 As shown, the image processing method provided in this disclosure includes the following steps S1100 to S1300.

[0026] Step S1100: Obtain the image sequence.

[0027] The image sequence includes multiple first images acquired sequentially by the image acquisition device, as well as metadata for each first image.

[0028] The metadata includes at least one of the following: feature values ​​of the motion mode of the image acquisition device when the first image was acquired, the image recording value of the first image, and the image subject identifier of the first image. Of course, the metadata may also include other data, such as the acquisition time of the first image, etc. The types of other data are not limited in this embodiment.

[0029] Among them, image recording value is used to quantify the degree to which an image in a corresponding scene is worth recording; the more worthwhile the image is to be recorded, the higher the image recording value. The feature values ​​of motion modes are, for example, one of stationary, jittering, uniform rotation, and accelerated rotation. The image subject identifier is used to identify the subject of the image and can be obtained from the first image through image analysis methods.

[0030] In one embodiment of this disclosure, multiple first images in an image sequence are acquired sequentially by an image acquisition device at fixed time intervals. Furthermore, metadata for each first image is extracted by the image acquisition device. In one example, the image acquisition device determines the image recording value based on the matching degree between the image content of the first image and the current scene. Of course, the image acquisition device may also determine the image recording value of the first image using other methods.

[0031] Step S1200: For each first image, identify whether the first image is a keyframe image based on the metadata of the first image.

[0032] In one embodiment of this disclosure, step S1200 can be specifically implemented by step S1210.

[0033] Step S1210: If the metadata of the first image satisfies any one of the first to third items below, the first image is determined to be a first keyframe image. Wherein, the keyframe image includes the first keyframe image.

[0034] The first step is to compare the image subject identifier corresponding to the first image with the image subject identifier of the second image, and determine that the image subject identifier corresponding to the first image and the image subject identifier of the second image are different. The second image is the frame image preceding the first image in the image sequence.

[0035] In one example, if the image sequence includes a first image P1, a first image P2, a first image P3, and a first image P4 in sequence, then when performing step S1200 on the first image P4, the second image is specifically the first image P3.

[0036] The second step involves comparing the feature values ​​of the motion pattern of the first image with those of the motion pattern of the second image to determine that the feature values ​​of the motion pattern of the first image are different from those of the motion pattern of the second image.

[0037] The third item is that the current image recording value of the first image is greater than the first preset image recording value, and the second preset image recording value is greater than the first preset image recording value.

[0038] Regarding the first item above, the image subject identifier corresponding to the first image is compared with the image subject identifier of the second image. If it is determined that the image subject identifier corresponding to the first image is different from the image subject identifier of the second image, it is determined that the image subject of the first image has changed and has high retention value. In this case, the first image is determined as the first keyframe image with high retention value.

[0039] Regarding the second item above, the feature values ​​of the motion mode of the first image and the feature values ​​of the motion mode of the second image are compared. If the feature values ​​of the motion mode of the first image and the feature values ​​of the motion mode of the second image are different, it indicates that the motion mode of the image acquisition device has changed and the first image has high retention value. At this time, the first image is determined as the first keyframe image with high retention value.

[0040] Regarding the third point above, if the current image recording value of the first image is greater than the first preset image recording value, it indicates that the image recording value of the first image is sufficiently high, and the first image has high retention value. In this case, the first image is determined as the first keyframe image with high retention value.

[0041] In one embodiment of this disclosure, if the metadata of the first image does not satisfy any of the first to third items below, the first image is determined to be a non-keyframe.

[0042] In one embodiment of this disclosure, if the metadata of the first image does not satisfy any of the first to third items mentioned above, the image processing method provided by this disclosure further provides the following second method for identifying whether the first image is a keyframe image. Specifically, if the metadata of the first image does not satisfy the first, second, and third items, the image processing method provided by this disclosure further includes the following steps S1220 and S1221.

[0043] Step S1220: Based on the first image and the third image, determine the image similarity between the first image and the third image, wherein the third image is the first keyframe image that is temporally closest to the first image.

[0044] Step S1221: If the image similarity is less than or equal to a preset similarity, determine the first image as the second keyframe image.

[0045] The keyframe image includes a second keyframe image. Image similarity is used to characterize the content similarity between the two images. The preset similarity is the maximum similarity between the first image and the second image when the content of the first image and the third image are significantly different.

[0046] In this disclosure, if it is determined from the metadata of the first image that the first image is not the first keyframe image, the similarity between the first image and the third image, that is, the similarity between the first image and the image content of the most recent first keyframe, is further used to determine whether it has retention value.

[0047] In one embodiment of this disclosure, image similarity can be obtained by calculating the cosine similarity or Euclidean distance between the simplified feature vectors of the first image and the third image. The simplified feature vectors can be obtained through color histograms, edge gradient histograms, neural networks, etc. Of course, image similarity can also be calculated in other ways, and this disclosure does not limit the specific implementation of the image similarity calculation method.

[0048] If the image similarity is less than or equal to the preset similarity, it means that the similarity between the first image and its closest first keyframe image is low, and the content difference between the first image and its closest first keyframe image is significant. In this case, the first image is determined to be the second keyframe image.

[0049] Through the above steps S1220 and S1221, this embodiment of the present disclosure, based on the identification of keyframes according to metadata, introduces image content similarity as a supplementary judgment, further identifying keyframe images with significant content differences but not judged as keyframe images according to metadata, forming a dual keyframe identification mechanism of "metadata priority and content fallback", avoiding the problem of images with recordable value being mistakenly deleted because the corresponding features of metadata are not significant.

[0050] Step S1300: If the first image is not a keyframe image, discard the first image; if the first image is a keyframe image, retain the first image.

[0051] If the first image is not a keyframe image, it means that the first image has no value for retention and is not worth keeping. In this case, the first image is discarded. Conversely, if the first image is a keyframe image, it is retained. That is, only the first image with retention value is retained. This avoids the problem of wasting storage space or consuming a lot of network resources by retaining the first image that has no retention value.

[0052] This disclosure provides an image processing method applied to an image processing device. The method includes: acquiring an image sequence, the image sequence including multiple first images sequentially acquired by an image acquisition device and metadata for each first image, the metadata including at least one of: feature values ​​of the motion mode of the image acquisition device when acquiring the first image, the image recording value of the first image, and the image subject identifier of the first image; for each first image, identifying whether the first image is a keyframe image based on the metadata of the first image; discarding the first image if it is not a keyframe image, and retaining the first image if it is a keyframe image. This method enables the retention of only first images with retention value. This avoids the problem of wasted storage space or high network resource consumption caused by retaining first images without retention value.

[0053] In one embodiment of this disclosure, the retention of the first image in step S1300 when the first image is a keyframe image can be specifically implemented through the following steps S1310 and S1320.

[0054] Step S1310: If the first image is a first keyframe image, compress the first image according to the first compression parameters.

[0055] Step S1320: If the first image is a second keyframe image, compress the first image according to the second compression parameters.

[0056] The compression quality of the first compression parameter is higher than that of the second compression parameter.

[0057] In one example of this disclosure, the compression quality of the first compression parameter is a JPEG quality factor of 85, and the compression quality of the second compression parameter is a JPEG quality factor of 75.

[0058] In this embodiment, the first keyframe image with relatively high retention value is compressed using a first compression parameter with relatively high compression quality, preserving more image details and ensuring the visual quality of key scenes. The second keyframe image with relatively low retention value is compressed using a second compression parameter with relatively low compression quality, saving more storage space while preserving the basic image content. Thus, differentiated compression based on the retention value of the keyframe images improves the intelligence of the image processing method provided in this disclosure.

[0059] In one embodiment of this disclosure, in order to further save storage space, the image processing method provided by this disclosure further includes the following step S1400 before the above step S1320.

[0060] Step S1400: Obtain the current remaining storage capacity value of the image processing device.

[0061] Based on the above step S1400, the above step S1320 is specifically implemented through the following step S1321.

[0062] Step S1321: If the current remaining storage capacity value is greater than or equal to the preset storage capacity value and the first image is the second keyframe image, compress the first image according to the second compression parameters.

[0063] Based on the above step S1321, the image processing method provided in this disclosure further includes the following step S1322.

[0064] Step S1322: When the current remaining storage capacity is less than the preset storage capacity and the first image is the second keyframe image, the first image is compressed according to the third compression parameter. The compression quality of the second compression parameter is higher than that of the third compression parameter.

[0065] The preset storage capacity value is the minimum storage capacity value when there is sufficient storage space, which can be set based on experience or experimentation.

[0066] In this embodiment, the current remaining storage capacity is first read. If the current remaining storage capacity is greater than or equal to a preset storage capacity and the first image is a second keyframe image, then the storage space is sufficient. In this case, the first image is compressed according to the second compression parameters. Conversely, if the current remaining storage capacity is less than the preset storage capacity and the first image is a second keyframe image, then the first image is compressed according to a third compression parameter with a compression quality lower than the second compression parameter. Thus, when storage space is insufficient, the compression rate of the second keyframe image is further increased, thereby saving storage space.

[0067] In one embodiment of this disclosure, an image sequence is acquired through the following steps S1110 to S1150. These steps S1110 to S1150 can be performed by an image acquisition device or an image processing device.

[0068] Step S1110: Obtain the current motion data collected by the motion sensor in the image acquisition device within a preset time period before the current moment.

[0069] In this embodiment of the disclosure, the image acquisition device includes a motion sensor and an image sensor.

[0070] In one embodiment of this disclosure, the motion sensor is exemplarily an inertial measurement unit (IMU). The preset duration can range from [10, 100] ms. In one example, the preset duration is exemplarily 50 ms. Of course, the preset duration can also be other values, and this disclosure does not limit the specific value of the preset duration.

[0071] Specifically, for step S1110 above, motion sensing data collected sequentially by the motion sensor within a preset time period before the current moment is read from the motion sensor and used as the current motion data, thereby realizing step S1110 above.

[0072] Step S1120: Determine the current image recording value and the current change in the viewing angle of the image sensor based on the current motion data.

[0073] In one embodiment of this disclosure, filtering may first be performed on the current motion data. Then, based on the filtered current motion data, the current image recording value and the current change in the viewpoint of the image sensor are determined.

[0074] In this embodiment of the disclosure, step S1120 specifically includes the following steps S1121 and S1122.

[0075] Step S1121: Determine the current change in the viewing angle of the image sensor based on the current motion data.

[0076] The current viewpoint change is used to describe the change in the viewpoint of the image sensor caused by the motion corresponding to the current motion data.

[0077] Furthermore, deep learning can be used to implement step S1121 above. Specifically, a training sample set is obtained, which includes multiple sets of training samples. Each set of training samples includes sample motion data and the corresponding sample viewpoint change. A preset neural network model is trained based on the training sample set to obtain a viewpoint change prediction model. On this basis, the current motion data is input into the viewpoint change prediction model, and the viewpoint change prediction model outputs the current viewpoint change. The method for acquiring the sample viewpoint change corresponding to the sample motion data is as follows: the image acquisition device is controlled to acquire an initial image; the image acquisition device is controlled to move according to the sample motion data and then acquire a post-motion image; the viewpoint shift between the initial image and the post-motion image is compared to obtain the viewpoint change, which is recorded as the sample viewpoint change corresponding to the sample motion data.

[0078] In another embodiment of this disclosure, when the current motion data includes the current angular velocity data in the image plane width direction, the above step S1121 is specifically implemented through the following steps S1121-1 to S1121-4.

[0079] Step S1121-1: Integrate the current angular velocity data to obtain the current yaw angle change in the coordinate system of the motion sensor.

[0080] In this embodiment of the disclosure, since the image acquisition device is typically in a left-right rotational motion—for example, a user wearing AI glasses turning their head left and right—to reduce the amount of data computation, this embodiment only acquires the current angular velocity data in the image plane width direction, i.e., the current angular velocity data of the yaw angle. Furthermore, the current angular velocity data in the image plane width direction is integrated to obtain the current yaw angle change of the image acquisition device in the coordinate system of the motion sensor.

[0081] Step S1121-2: Convert the current yaw angle change in the coordinate system of the motion sensor into the current yaw angle change in the coordinate system of the image sensor.

[0082] Step S1121-3: Determine the current pixel displacement in the image plane width direction based on the current yaw angle change in the image sensor coordinate system.

[0083] In one embodiment of this disclosure, based on the camera focal length model, the current yaw angle change in the image sensor coordinate system is converted into the current pixel displacement in the image plane width direction using the following formula.

[0084] (Formula 1) in, This represents the current pixel displacement along the width direction of the image plane. The focal length of the image sensor; This represents the current change in yaw angle in the image sensor coordinate system.

[0085] Thus, through the above steps S1121-3, the current yaw angle change in the image sensor coordinate system is converted into pixel displacement in the image plane width direction.

[0086] Step S1121-4: Based on the current pixel displacement and the image plane width value corresponding to the image sensor, determine the current viewing angle change percentage in the image plane width direction, and use it as the current viewing angle change of the image sensor.

[0087] Specifically, the above steps S1121-4 are implemented through the following formula two.

[0088] (Formula 2) in, This represents the percentage change in the current viewing angle along the width of the image plane. This is the image plane width value corresponding to the image sensor, which is determined by the field of view (FOV) of the image sensor.

[0089] Through the above steps S1121-1 to S1121-4, the current angular velocity data can be converted into a percentage change in the viewing angle relative to the width of the entire image plane.

[0090] Of course, if the image acquisition device is rotating in the left-right and up-down directions, such as when a user wearing AI glasses turns their head left-right and up-down, then the current motion data includes the current angular velocity data in the image plane height direction in addition to the current angular velocity data in the image plane width direction. Based on this, the current pixel displacement in the image plane width direction and the current pixel displacement in the image plane height direction can be determined using a method similar to steps S1121-1 to S1121-3 described above. Further, the product of the current pixel displacement in the image plane width direction and the current pixel displacement in the image plane height direction is recorded as the current viewing angle change of the image sensor. Further, the ratio between the current viewing angle change of the image sensor and the total number of pixels in the image plane is recorded as the current viewing angle change of the image sensor. It should be noted that whether this method or the method described in steps S1121-1 to S1121-4 is used depends on the usage scenario and / or power consumption requirements of the image acquisition device.

[0091] Step S1122: Determine the current image recording value based on the current motion data.

[0092] In one embodiment of this disclosure, step S1122 can be specifically implemented by the following steps S1122-1 and S1122-2.

[0093] Step S1122-1: Based on the current motion data, determine the current feature value of at least one motion feature of the image acquisition device.

[0094] Among them, at least one of the following motion characteristics is motion intensity, motion smoothness, and motion pattern.

[0095] Continuing with the example of current angular velocity data in the image plane width direction, for motion intensity, the average value of the angular velocities corresponding to the current angular velocity data within a preset time period before the current moment can be determined as the motion characteristic value of motion intensity. The larger the average value, the faster the image acquisition device rotates, i.e., the greater the motion intensity. Furthermore, for motion smoothness, the standard deviation of the angular velocities corresponding to the current angular velocity data within a preset time period before the current moment can be determined as the motion characteristic value of motion smoothness. The larger the standard deviation, the less smooth the rotation of the image acquisition device. And, for motion mode, if the average value is less than or equal to a first threshold and the standard deviation is less than or equal to a second threshold, the motion characteristic value of motion mode is determined to be stationary. If the standard deviation is less than or equal to the second threshold, but the average value is greater than the first threshold, the motion characteristic value of motion mode is determined to be uniform rotation. If the standard deviation is greater than or equal to a third threshold and the average value is greater than or equal to a fourth threshold, the motion characteristic value of motion mode is determined to be accelerated rotation. If the aforementioned standard deviation is greater than or equal to the third threshold, but the aforementioned average value is less than the fourth threshold, then the motion characteristic value of the motion pattern is determined to be jitter.

[0096] The fourth threshold is greater than the first threshold, the third threshold is greater than the second threshold, and the first, second, third, and fourth thresholds are determined experimentally. The first threshold is the maximum value of the aforementioned average when the image acquisition device is stationary, the second threshold is the maximum value of the aforementioned standard deviation when the electronic device is stationary, the third threshold is the minimum value of the aforementioned standard deviation when the image acquisition device is accelerating or shaking, and the fourth threshold is the minimum value of the aforementioned average when the image acquisition device is accelerating and rotating.

[0097] Step S1122-2: Determine the current image recording value based on the current feature value of at least one motion feature.

[0098] In this embodiment, the feature values ​​of motion features can reflect the user's visual attention switching behavior. However, visual attention switching behavior can indicate whether the current scene is worth recording. Therefore, the value of image recording can be quantified through the feature values ​​of motion features.

[0099] The current image recording value is used to quantify the degree to which an image in the current scene is worth recording; the more worthwhile an image is to be recorded, the greater the current image recording value.

[0100] In one embodiment of this disclosure, when there is only one motion feature, the current feature value of that motion feature is directly normalized, and the normalized current feature value is recorded as the current image recording feature value. For example, if the motion feature value corresponding to motion mode F_c is uniform rotation and the corresponding normalized motion feature value is 0.7, the current image recording feature value is 0.7. If the motion feature value corresponding to motion mode F_c is stationary and the corresponding normalized motion feature value is 0, the current image recording feature value is 0. If the motion feature value corresponding to motion mode F_c is accelerated rotation and the corresponding normalized motion feature value is 0.9, the current image recording feature value is 0.9. If the motion feature value corresponding to motion mode F_c is jitter and the corresponding normalized motion feature value is 0.2, the current image recording feature value is 0.2.

[0101] In one embodiment of this disclosure, when the motion characteristics include at least two of motion intensity, motion smoothness, and motion pattern, the above step S1122-2 can be specifically implemented by the following steps S1122-21 and S1122-22.

[0102] Step S1122-21: Perform normalization processing on the current feature value of each motion feature to obtain the current normalized feature value of each motion feature.

[0103] Through the above steps S1122-21, motion characteristic values ​​of different dimensions are uniformly mapped to the [0,1] interval, which facilitates subsequent weighted summation.

[0104] Steps S1122-22: According to the target weight coefficient combination, the weighted sum of the current normalized feature values ​​of each motion feature is determined as the current image recording value.

[0105] The target weight coefficient combination includes a weight coefficient for each motion feature. In one embodiment of this disclosure, the target weight coefficient combination is a default combination of weight coefficients. For example, in the target weight coefficient combination, the weight coefficient w3=0.5 for motion intensity F_a, the weight coefficient w2=-0.3 for motion stability F_b, and the weight coefficient w1=1 for motion mode F_c.

[0106] Taking motion features including motion intensity, motion stability, and motion pattern as an example, if the motion feature value corresponding to motion intensity F_a is 20° / s and the corresponding normalized motion feature value is 0.4, the motion feature value corresponding to motion stability F_b is 2° / s and the corresponding normalized motion feature value is 0.1, and the motion feature value corresponding to motion pattern F_c is uniform rotation and the corresponding normalized motion feature value is 0.7, and the target weight coefficient combination is that the weight coefficient of motion intensity F_a is w3=0.5, the weight coefficient of motion stability F_b is w2=-0.3, and the weight coefficient of motion pattern F_c is w1=1, then the current image recording value S=1*0.7+(-0.3)*0.1+0.5*0.4=0.7-0.03+0.2=0.87.

[0107] Taking the motion mode as a shake as an example, the normalized feature value corresponding to the motion stability F_b is relatively large, but the weight coefficient of motion stability F_b is w2=-0.3, which has a large negative contribution, thus lowering the current image recording value. This can effectively suppress false image acquisition caused by shake. Continuing with the motion mode as an accelerated rotation example, the current normalized feature value (0.9) corresponding to the motion mode is very large. Although the weight coefficient of motion stability F_b is negative, the current image recording value can still easily exceed the preset image recording value. Based on this, it is possible to ensure the capture of fast movements.

[0108] Step S1130: When the current image recording value is greater than or equal to the second preset image recording value, or when the current viewpoint change is greater than or equal to the preset viewpoint change, control the image sensor in the image acquisition device to acquire an image and obtain the first image.

[0109] Among them, the value of the first preset image recording is greater than the value of the second preset image recording.

[0110] The second preset image recording value is the minimum image recording value when the image is highly likely to be recorded, and it can be determined based on experience, experiments, or scenarios.

[0111] If the current image recording value is greater than or equal to the preset image recording value, it indicates that the image is likely worth recording. At this point, the image sensor is controlled to acquire an image, and the image acquired by the image sensor is recorded as the first image.

[0112] Furthermore, the preset viewing angle change amount is the minimum viewing angle change amount of the image sensor when the image content is highly likely to change significantly. In one embodiment of this disclosure, the preset viewing angle change amount can range from [20%, 50%]. In one example, the preset viewing angle change amount is 40%.

[0113] If the change in the current viewing angle is greater than or equal to the preset change in viewing angle, it indicates that the image content has likely changed significantly. At this point, to avoid missing key moments during the viewing angle switching process, the image sensor is controlled to acquire an image, and the image acquired by the image sensor is recorded as the first image.

[0114] Corresponding to step S1130 above, if the current image recording value is less than the first preset image recording value and the current viewpoint change is less than the preset viewpoint change, then the image sensor is not controlled to acquire images.

[0115] It should be noted that, as the current moment is updated, the image acquisition device or image processing device executes the above step S1110 in real time.

[0116] As discussed above, compared to traditional techniques that control image sensor acquisition at set time intervals, this method avoids image acquisition when the change in the current viewing angle is less than a preset change in viewing angle (i.e., the image content has not changed significantly). This avoids the problems of image sensors acquiring highly overlapping images at set time intervals, which would consume electronic device storage space and increase the burden on subsequent image processing. Furthermore, this method acquires images only when the change in the current viewing angle is greater than or equal to a preset change in viewing angle (i.e., the image content is likely to change significantly). This also avoids the problem of image sensors missing key moments during field-of-view switching when acquiring images at set time intervals, resulting in discontinuous recording and an inability to fully reconstruct the scene.

[0117] Furthermore, the image sensor is controlled to acquire images only when the current image recording value is greater than or equal to the first preset image recording value, that is, when the current image recording value is sufficient. This avoids to some extent the problem of the image sensor recording images without image recording value when it acquires images at set time intervals, which would occupy the storage space of the image acquisition device and increase the pressure on subsequent image processing. It also avoids the problem of the image sensor missing key images with image recording value when acquiring images at set time intervals, which would cause gaps in the recorded content and make it impossible to fully restore the scene.

[0118] It should be noted that due to the complexity of the scene, as well as issues such as the recording value of the second preset image and the accuracy of the preset viewpoint change, there may still be content similarity between the first images acquired sequentially by the image sensor.

[0119] In one embodiment of this disclosure, step S1500 may be included before step S1130 described above.

[0120] Step S1500: Obtain motion avoidance feature values.

[0121] Motion avoidance features are used to describe the types of motion features that tend to be avoided during image acquisition, thus eliminating invalid triggering scenarios. An exemplary motion avoidance feature is a jittery motion pattern, but other motion avoidance features are also possible; this disclosure does not limit the specific form of the motion avoidance feature.

[0122] In one embodiment of this disclosure, motion avoidance feature values ​​can be pre-input by the user into the image acquisition device or image processing device.

[0123] In another embodiment of this disclosure, the motion avoidance feature value can be learned by the image acquisition device or the image processing device from the image processing operations performed by the user on images historically acquired by the image sensor. Specifically, step S1500 is implemented through steps S1510 and S1511 described below.

[0124] Step S1510: Receive image processing operations for the historical image set, including deletion and / or editing.

[0125] Step S1511: In response to the image processing operation, identify and acquire motion avoidance feature values.

[0126] In this embodiment of the disclosure, the historical image set includes images acquired by the image sensor at historical moments. When a user views images in the historical image set, they typically perform image processing operations. For example, they may delete images that do not meet their needs or edit images to better suit their requirements. Based on this, the electronic device can receive the user's image processing operations on the historical image set.

[0127] It is understood that image processing operations performed by a user on images in a historical image set can reflect motion avoidance feature values. In one embodiment of this disclosure, an AI model can be used to identify the image corresponding to the image processing operation in order to extract motion avoidance feature values.

[0128] As can be seen from the above steps S1510 and S1511, by monitoring the user's deletion and editing operations on images acquired at historical moments, the system automatically learns and updates the user's motion avoidance feature values. This achieves the automatic acquisition of personalized motion avoidance feature values ​​without requiring active configuration by the user, thereby improving the intelligence of image acquisition and the user experience.

[0129] Based on the above step S1500, the above step S1130 is specifically implemented through the following step S1131.

[0130] Step S1131: When the current image recording value is greater than or equal to the preset image recording value, and the feature value of at least one motion feature does not meet the motion avoidance feature value, control the image sensor to acquire an image.

[0131] In this embodiment, the system determines whether the current image recording value is greater than or equal to a preset image recording value, and whether the feature value of at least one motion feature meets the motion avoidance feature value. If the current image recording value is greater than or equal to the preset image recording value, and the feature value of at least one motion feature does not meet the motion avoidance feature value, then the image sensor is controlled to acquire an image. This further eliminates motion scenarios that the user does not want to trigger, improves the triggering accuracy of the image sensor, and reduces the acquisition of invalid images.

[0132] Step S1140: Extract the metadata of the first image.

[0133] In this embodiment of the disclosure, the image record value of the first image in the metadata of the first image is specifically calculated by the current image record value in step S1122-2 above.

[0134] Step S1150: Generate an image sequence based on the first image and its corresponding metadata acquired sequentially by the image acquisition device.

[0135] like Figure 2 As shown, this disclosure also provides an image processing apparatus 200, applied to an image processing device, comprising: The acquisition module 210 is used to acquire an image sequence, the image sequence including multiple frames of first images acquired sequentially by the image acquisition device and metadata of each first image, the metadata including at least one of the following: feature value of the motion mode of the image acquisition device when the first image is acquired, image recording value of the first image, and image subject identifier of the first image; The recognition module 220 is used to identify, for each of the first images, whether the first image is a keyframe image based on the metadata of the first image; The processing module 230 is configured to discard the first image if the first image is a non-keyframe image, and retain the first image if the first image is a keyframe image.

[0136] In one embodiment of this disclosure, the identification module 220 is specifically configured to determine that the first image is a first keyframe image when the metadata of the first image satisfies any one of the first to third items below, wherein the keyframe image includes the first keyframe image. The first step involves comparing the image subject identifier corresponding to the first image with the image subject identifier of the second image to determine that the image subject identifier corresponding to the first image and the image subject identifier of the second image are different, and the second image is the frame image preceding the first image in the image sequence. The second step is to compare the feature values ​​of the motion pattern of the first image with the feature values ​​of the motion pattern of the second image to determine that the feature values ​​of the motion pattern of the first image are different from the feature values ​​of the motion pattern of the second image. Thirdly, the image recording value of the first image is greater than the first preset image recording value.

[0137] In one embodiment of this disclosure, the identification module 220 is further configured to determine the image similarity between the first image and the third image based on the first image and the third image, wherein the third image is the first keyframe image that is temporally closest to the first image; If the image similarity is less than or equal to a preset similarity, the first image is determined to be the second keyframe image, and the keyframe image includes the second keyframe image.

[0138] In one embodiment of this disclosure, the processing module 230 is specifically configured to compress the first image according to the first compression parameters when the first image is a first keyframe image; If the first image is a second keyframe image, the first image is compressed according to the second compression parameters; The compression quality of the first compression parameter is higher than that of the second compression parameter.

[0139] In one embodiment of this disclosure, the acquisition module 210 is specifically used to acquire the current remaining storage capacity value of the remaining storage capacity of the image processing device; The processing module 230 is specifically configured to compress the first image according to a second compression parameter when the current remaining storage capacity value is greater than or equal to a preset storage capacity value and the first image is a second keyframe image; and to compress the first image according to a third compression parameter when the current remaining storage capacity value is less than the preset storage capacity value and the first image is a second keyframe image, wherein the compression quality of the second compression parameter is higher than the compression quality of the third compression parameter.

[0140] In one embodiment of this disclosure, the image sequence is obtained according to the following steps: Acquire the current motion data collected by the motion sensor in the image acquisition device within a preset time period before the current moment; Based on the current motion data, determine the current image recording value and the current change in viewing angle of the image sensor in the image acquisition device; When the current image recording value is greater than or equal to the second preset image recording value, or when the current viewpoint change is greater than or equal to the preset viewpoint change, the image sensor is controlled to acquire an image to obtain a first image, wherein the first preset image recording value is greater than the second preset image recording value. Extract the metadata of the first image; An image sequence is generated based on the first image and its corresponding metadata acquired sequentially by the image acquisition device.

[0141] In one embodiment of this disclosure, the current motion data includes current angular velocity data in the image plane width direction, and determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Integrating the current angular velocity data yields the current yaw angle change in the coordinate system of the motion sensor. Convert the current yaw angle change in the coordinate system of the motion sensor to the current yaw angle change in the coordinate system of the image sensor. Based on the current change in yaw angle in the image sensor coordinate system, determine the current pixel displacement in the image plane width direction; Based on the current pixel displacement and the image plane width value corresponding to the image sensor, the percentage change in the current viewing angle in the image plane width direction is determined as the current viewing angle change of the image sensor.

[0142] In one embodiment of this disclosure, determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Based on the current motion data, determine the current feature value of at least one motion feature of the image acquisition device; The current image recording value is determined based on the current feature value of the at least one motion feature.

[0143] This disclosure also provides an electronic device, which includes any of the image processing devices 200 provided in the above-described device embodiments.

[0144] like Figure 3As shown, this disclosure also provides another electronic device 300, which includes a memory 310 and a processor 320. The memory 310 is used to store computer instructions, and the processor 320 is used to call the computer instructions from the memory 310 to execute any of the image processing methods provided in the above method embodiments.

[0145] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the image processing methods provided in the above-described method embodiments.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0147] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0148] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0150] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the embodiments of this disclosure.

[0151] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0155] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An image processing method, characterized in that, Applied to an image processing device, the method includes: An image sequence is acquired, the image sequence including multiple frames of first images acquired sequentially by an image acquisition device and metadata of each first image, the metadata including at least one of the following: feature value of the motion mode of the image acquisition device when the first image is acquired, image recording value of the first image, and image subject identifier of the first image; For each of the first images, identify whether the first image is a keyframe image based on the metadata of the first image; If the first image is not a keyframe image, discard the first image; if the first image is a keyframe image, retain the first image.

2. The method according to claim 1, characterized in that, The step of identifying whether the first image is a keyframe image based on the metadata of the first image includes: If the metadata of the first image satisfies any one of the first to third items below, the first image is determined to be a first keyframe image, and the keyframe image includes the first keyframe image. The first step involves comparing the image subject identifier corresponding to the first image with the image subject identifier of the second image to determine that the image subject identifier corresponding to the first image and the image subject identifier of the second image are different, and the second image is the frame image preceding the first image in the image sequence. The second step is to compare the feature values ​​of the motion pattern of the first image with the feature values ​​of the motion pattern of the second image to determine that the feature values ​​of the motion pattern of the first image are different from the feature values ​​of the motion pattern of the second image. Thirdly, the image recording value of the first image is greater than the first preset image recording value.

3. The method according to claim 2, characterized in that, If the metadata of the first image does not satisfy the first, second, and third items, the method further includes: Based on the first image and the third image, determine the image similarity between the first image and the third image, wherein the third image is the first keyframe image that is temporally closest to the first image; If the image similarity is less than or equal to a preset similarity, the first image is determined to be the second keyframe image, and the keyframe image includes the second keyframe image.

4. The method according to claim 3, characterized in that, The step of preserving the first image when the first image is a keyframe image includes: If the first image is a first keyframe image, the first image is compressed according to the first compression parameters; If the first image is a second keyframe image, the first image is compressed according to the second compression parameters; The compression quality of the first compression parameter is higher than that of the second compression parameter.

5. The method according to claim 4, characterized in that, Before compressing the first image according to the second compression parameters when the first image is a second keyframe image, the method further includes: Obtain the current remaining storage capacity value of the image processing device; When the first image is a second keyframe image, compressing the first image according to the second compression parameters includes: If the current remaining storage capacity value is greater than or equal to the preset storage capacity value and the first image is a second keyframe image, the first image is compressed according to the second compression parameters. The method further includes: When the current remaining storage capacity is less than the preset storage capacity and the first image is a second keyframe image, the first image is compressed according to the third compression parameter, where the compression quality of the second compression parameter is higher than the compression quality of the third compression parameter.

6. The method according to claim 2, characterized in that, The image sequence is obtained according to the following steps: Acquire the current motion data collected by the motion sensor in the image acquisition device within a preset time period before the current moment; Based on the current motion data, determine the current image recording value and the current change in viewing angle of the image sensor in the image acquisition device; When the current image recording value is greater than or equal to the second preset image recording value, or when the current viewpoint change is greater than or equal to the preset viewpoint change, the image sensor is controlled to acquire an image to obtain a first image, wherein the first preset image recording value is greater than the second preset image recording value. Extract the metadata of the first image; An image sequence is generated based on the first image and its corresponding metadata acquired sequentially by the image acquisition device.

7. The method according to claim 6, characterized in that, The current motion data includes current angular velocity data in the image plane width direction. Determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Integrating the current angular velocity data yields the current yaw angle change in the coordinate system of the motion sensor. Convert the current yaw angle change in the coordinate system of the motion sensor to the current yaw angle change in the coordinate system of the image sensor. Based on the current change in yaw angle in the image sensor coordinate system, determine the current pixel displacement in the image plane width direction; Based on the current pixel displacement and the image plane width value corresponding to the image sensor, the percentage change in the current viewing angle in the image plane width direction is determined as the current viewing angle change of the image sensor.

8. The method according to claim 6, characterized in that, The step of determining the current image recording value and the current viewpoint change of the image sensor in the image acquisition device based on the current motion data includes at least: Based on the current motion data, determine the current feature value of at least one motion feature of the image acquisition device; The current image recording value is determined based on the current feature value of the at least one motion feature.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.